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Foreword 


The  Interactive  Electronic  Technical  Manual/ Automated  Classroom  (lETM/AC)  Technology 
project  was  sponsored  by  the  Bureau  of  Naval  Personnel  under  Program  Element  0603707N.  This 
survey  of  lETMs  used  for  training  and  education  was  completed  under  the  Navy  Personnel 
Research  and  Development  Center  Contract  N66001-91-D-9502. 

The  survey  and  analysis  was  conducted  of  practices  in  the  design,  development,  and 
application  of  education  and  training  that  incorporates  interactive,  hypermedia-based  and  lETM- 
like  documents.  This  report  is  directed  to  training  communities  and  program  managers  who  may 
be  developing  or  implementing  lETMs. 


J.  C.  McLACHLAN 
Director,  Classroom  and  Afloat  Training 
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Summary 


Background 

Navy  Personnel  Research  and  Development  Center  (NPRDC),  working  with  the  Naval  Sea 
Systems  Command  (NAVSEA)  and  Chief  of  Naval  Education  and  Training  (CNET),  completed  a 
program  of  research,  development,  and  evaluation  for  automating  classroom  activities  in  which 
interactive  electronic  technical  manuals  (lETMs)  are  used.  Advances  in  technology  based  on 
electronic  data  conversion,  storage,  transfer  and  standards  have  provided  the  Navy  with  the 
opportunity  to  investigate  the  design  and  development  of  prototype  automated  classrooms.  This 
new  learning  environment  may  provide  significant  advances  in  instructional  development  and 
presentation  effectiveness,  as  well  as  efficiencies  in  costs. 

Objective 

The  objective  of  this  survey  and  analysis  was  to  identify  lessons  that  have  been  learned, 
problems  and  issues  that  have  been  identified,  and  successful  techniques  and  tools  that  have  been 
developed.  The  goal  was  to  report  these  findings  and  thereby  enhance  future  integration  of  lETM 
technology  and  training. 

Approach 

The  topic  of  lETMs  and  their  integration  into  education  and  training  is  relatively  broad  and  can 
be  technically  complex.  It  is  expected  that  the  readers  of  this  report  will  be  a  diverse  group  with  a 
broad  spectrum  of  backgrounds.  Therefore,  an  overview  of  lETMs  is  provided.  Applications 
surveyed  included  such  instructional  settings  as  traditional  classroom  instruction,  virtual 
classrooms,  and  on-demand  and  just-in-time  instruction.  The  survey  covered  both  Navy  and  non- 
Navy  applications.  Non-Navy  activities  included  other  military,  government,  industry,  and  higher 
education  organizations. 

Conclusions 

lETMs  impact  not  only  how  people  access,  navigate,  and  use  technical  documents  but  also 
impact  the  job  definition,  aspects  of  training,  and  the  organizations  responsible  for  job  functions, 
training,  and  technical  documents.  The  integration  of  lETM  technology  into  training  and  education 
requires  a  look  at  the  roles  and  functions  of  organizations. 

The  introduction  of  computers  in  classrooms  and  laboratories  to  support  instruction  using 
lETMs  parallels  the  movement  to  implement  electronic  classrooms.  The  current  model  of  the 
classroom  being  instructor-led  with  group-paced  instruction  is  now  being  transferred  to  the  lETM- 
based  electronic  classrooms.  The  classroom  will  continue  to  change  as  its  functionality  is  enhanced 
by  new  technology. 

lETM-based  training  may  result  in  fewer  courses  and  less  training  time.  An  option  being 
considered  is  to  move  advanced  training  out  of  the  classroom  and  into  the  field  with  smart  lETMs 
and  computer-based  training  that  can  be  accessed  as  needed.  For  example,  the  concept  of  just-in- 


time  training  with  a  performance  support  system  may  be  appropriate  for  areas  such  as  aircraft 
maintenance. 

lETM  technology  may  redefine  jobs  and  training  requirements.  Several  organizations 
expressed  strong  preferences  for  the  capability  to  support  the  technician  through  more  use  of 
lETMs  as  expert  job  aids  and  less  use  of  training.  The  more  intelligence  built  into  an  lETM,  the 
more  support  for  equipment  operators  and  maintainers. 

Given  the  cost  of  data  development  for  both  technical  manuals  and  training,  and  the 
commonality  of  information  used  in  both  applications,  it  seems  reasonable  to  consider  integrating 
the  databases  to  support  diverse  applications. 

Educational  organizations  were  found  to  be  very  active  in  producing  and  using  hypertext/ 
hypermedia  documents.  Often  the  goal  is  to  teach  students  how  to  access  information,  determine 
relationships,  use  information  to  solve  problems,  and  develop  multiple  perspectives  on  the 
information.  lETM  technologies  can  be  used  to  support  such  goals  in  education. 


viii 


r 


Contents 

Page 

Section  One:  Introduction .  i 

1.1  Purpose . ^ 

1.2  Organization  of  the  Report .  ^ 

1.3  Evolutions .  ^ 

Section  Two:  Nature  of  Interactive  Electronic  Technical 
Manuals .  ^ 

2.1  Overview  of  lETMs . ^ 

2.2  Need  for  . .  ^ 

2.3  lETM  Technology .  ^ 

2.4  lETM  Classification  System . .  H 

2.5  lETM  Development  Process . 

2.5.1  Mature  System  lETM  Development .  14 

2.5.2  Authoring  lETMs  for  New  Systems .  16 

Section  Three:  Current  lETM  and  Other  SGML  Based 
Implementations . 21 

3.1  Military  lETMs . - . 

3.2  Other  Government  and  Industry  SGML  Documents .  29 

3.3  SGML  Applications  in  Higher  Education .  33 

Section  Four:  Integration  of  lETMs  with  Training  and 
Education .  39 

4.1  lETM  Training  Settings .  41 

4.1.1  lETMs  in  the  Classroom .  41 

4.1.2  lETMs  in  On-Demand  and  Just-In-Time  Training .  43 

4. 1 .3  lETMs  as  Expert  Job  Aids .  44 

4.2  Implications  for  the  Organizations .  44 

4.2. 1  Coordinated  Definition  and  Development  of  lETMs .  45 

4.2.2  Course  Management .  46 

4.2.3  Planning .  46 

4.2.4  Life  Cycle  Support .  47 

4.2.5  Instructor,  Student,  and  Staff  Training .  48 

4.3  Implications  for  Embedding  Training  Content  in  lETMs .  51 

4.4  Integration  ofTraining  Data  with  Technical  Information .  52 

4.4.1  Use  of  lETMs  as  Training  Media .  52 

4.4.2  Use  of  lETM  Data  as  a  Source  for  Developing  Training .  52 


IX 


4.4.3  Common  Data  Elements  and  Integrated  Databases .  53 

4.5  Impact  on  Education . 54 

Section  Five:  Differences  Between  Job  Performance  Support 
Documentation  and  Training .  57 

5.1  Hypertext  and  Learning . . .  59 

5.2  Animation  and  Learning .  62 

5.3  Motion  Video  and  Learning . 64 

Section  Six:  Conclusions .  67 

6. 1  Preparing  and  Supporting  the  Training  Settings  for  lETMs .  69 

6.2  Changing  Nature  of  lETM  electronic  Classrooms .  69 

6.3  lETMs  in  On-Demand  and  Just-In-Time  Training .  69 

6.4  lETMs  as  Expert  Job  Aids .  70 

6.5  Implications  for  the  Organizations .  70 

6.6  lETM  Data  as  a  Source  for  Developing  Training .  70 

6.7  Common  Data  Elements  and  Integrated  Databases .  70 

6.8  lETM  Technology  in  Education .  71 

6.9  Embedding  Training  Within  lETMs .  71 

6.10  Beyond  lETMs  and  SGML .  72 

Section  Seven:  Standards  and  Specifications .  73 

Section  Eight:  References .  77 

Section  Nine:  Distribution  List . 83 


List  of  Tables 

1.  Tri-Service  Definitions  for  Classes  of  Electronic  Technical  Manuals .  12 

2.  Some  Military  lETMs  Currently  in  the  Field  or  Under  Development .  24 

3.  SGML  Projects  in  Other  Government  Agencies  and  Industry .  30 

4.  Higher  Education  Applications  of  SGML  Technology .  35 


X 


Section  One 
Introduction 


1.0  Introduction 


The  Navy  Personnel  Research  and  Development  Center  (NPRDC),  working  with  the  Naval 
Sea  Systems  Command  (NAVSEA)  and  the  Chief  of  Naval  Education  and  Training  (CNET),  is 
currently  conducting  a  program  of  research,  development,  and  evaluation  for  automating 
classroom  activities  in  which  electronic  technical  manuals  are  used.  Advances  in  technology  based 
on  electronic  data  conversion,  storage,  transfer  and  standards  have  provided  the  Navy  with  the 
opportunity  to  investigate  the  design  and  development  of  prototype  instructor  and  student  stations 
that  may  provide  not  only  significant  advances  in  instructional  development  and  presentation 
effectiveness,  but  efficiencies  in  costs.  The  advanced  hardware  and  software  technologies  which 
have  emerged  to  make  this  possible  include:  (1)  high  quality  digital  multimedia,  (2)  hypertext/ 
hypermedia  to  allow  greater  use  of  information  in  all  forms,  (3)  the  capability  to  store  digital  media 
materials  with  greater  flexibility  and  less  costs  through  storage  technology  such  as  CD-ROM,  and 
(4)  transmission  of  multimedia  through  local,  wide  area,  and  worldwide  networks. 

Navy  logistics  and  documentation  agencies  also  have  noted  the  advancing  technologies  and 
applied  these  capabilities  to  support  Navy  operators  and  maintenance  technicians.  A  primary  effort 
has  been  focused  on  the  design  and  development  of  Interactive  Electronic  Technical  Manuals 
(lETMs).  lETMs  are  intended  to  solve  a  number  of  problems  including:  (1)  the  information 
intensive  job  support  required  by  today’s  complex  equipment  and  systems,  (2)  the  paper  intensive 
representation  of  that  information  (with  associated  problems  of  storage  and  retrieval  both  ashore 
and  afloat),  and  (3)  the  access  to  relevant  information  in  a  timely,  on-demand  manner.  lETMs,  as 
intelligent  electronic  documents,  are  specifically  designed  to  address  these  problems  and  thereby 
improve  job  performance  while  reducing  costs  of  operation  and  maintenance. 

For  the  military,  there  is  a  natural  marriage  of  the  two  functions,  technical  documentation 
and  training,  based  on  the  common  advanced  technology  that  can  enhance  both.  As  lETMs  become 
more  common  they  will  be  used  in  training  classrooms  and  field  training,  and  in  turn  they  will  be 
supplemented  by  training  materials.  There  is  a  clear  need  to  include  documentation  and  training 
support  plans  early  in  the  system  acquisition  process. 

1.1  Purpose 

A  survey  and  analysis  was  conducted  of  practices  in  the  design,  development,  and 
application  of  education  and  training  that  incorporates  interactive,  hypermedia-based  and  lETM- 
like  documents.  The  purpose  of  the  survey  was  to  identify  lessons  that  have  been  learned,  problems 
and  issues  that  have  been  identified,  and  successful  techniques  and  tools  that  have  been  developed. 
Our  goal  was  to  report  these  and  thereby  enhance  future  integration  of  lETM  technology  and 
training.  The  applications  surveyed  included  such  instructional  settings  as  traditional  classroom 
instruction,  virtual  classrooms,  and  on-demand  and  just-in-time  instruction.  The  survey  included 
both  Navy  and  non-Navy  applications.  Non-Navy  activities  included  other  military,  government, 
industry  and  higher  education  organizations.  The  survey  was  conducted  through  site  visits,  phone 
calls,  and  reviews  of  documentation. 
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1.2  Organization  of  the  Report 

The  topic  of  lETMs  and  their  integration  into  education  and  training  is  relatively  large  and 
can  be  technically  complex.  It  is  expected  that  the  readers  of  this  report  will  be  a  diverse  group 
including  education  and  training  specialists,  computer  and  communication  experts,  career  Navy 
personnel  from  a  broad  spectrum  of  backgrounds,  weapon  system  acquisition  and  support 
personnel,  and  both  traditional  and  electronic  documentation  authorities. 

For  these  reasons: 

•  Section  2  discusses  the  nature  of  lETMs.  Although  the  primary  objective  of  the  report 
is  to  address  the  integration  of  lETM  technology  and  training,  it  is  necessary  to  discuss 
aspects  of  lETM  technology  to  place  the  implications  for  training  in  perspective. 
Those  readers  who  are  well  acquainted  with  lETM  technology  can  skip  or  skim  Sec¬ 
tion  2.  Those  readers  who  are  new  to  lETM  technology  should  read  it  initially  for  a 
cursory  understanding  only.  In  later  sections,  we  make  specific  references  so  readers 
can  make  the  link  between  the  lETM  technology  and  the  training  implications. 

•  Section  3  describes  current  and  planned  applications  covered  in  the  survey  of  lETM 
practices. 

•  Section  4  discusses  the  integration  of  lETMs  with  training  and  education  on  the  basis 
of  both  currently  implemented  or  planned  applications  included  in  the  survey.  This 
section  describes  the  many  considerations,  including  the  evolving  decision  and  plan¬ 
ning  techniques,  that  need  to  be  addressed  in  integrating  lETMs  and  training. 

•  Section  5  describes  the  differences  between  job  performance  support  documentation 
and  training.  This  section  highlights  some  very  important  differences  between  the  two 
applications.  We  argue  that  although  a  great  deal  of  common  data  is  utilized,  the  dif¬ 
ferences  in  objectives  are  fundamental  to  how  respective  electronic  documents  in 
these  applications  are  designed,  presented,  maintained,  managed,  and  used. 

•  Section  6  summarizes  the  lessons  learned,  issues  and  problems,  and  successful  trends. 
The  conclusions  recommend  some  points  of  integration  for  the  future. 

•  Section  7  presents  the  references  to  other  documents  included  in  this  report. 

1.3  Evolutions 

The  evolutionary  nature  of  lETM  technology,  goals,  and  developmental  aspects  are  evident 
throughout  this  report.  Likewise,  the  integration  of  lETMs  with  education  and  training  is  also 
evolutionary  and  discussed  as  such.  It  is  important  to  recognize  the  evolutionary  nature  of  these 
applications  of  technology  as  well  as  the  evolutionary  nature  of  the  technologies  themselves.  Not 
to  see  the  evolutions  in  perspective  could  mean  missing  the  larger  picture  of  where  we  are  headed 
in  the  logistics,  documentation,  and  training  communities  and  how  we  should  plan  for  a  reasonably 
solid  infrastructure  upon  which  to  complete  our  respective  taskings.  This  is  a  matter  of  importance 
not  only  to  Navy  operational  management,  but  to  system  commands,  research  and  development 
laboratories,  and  education  and  training  commands. 
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Section  Two 


Nature  of  Interactive  Electronic  Technical  Manuals 


2.0  Nature  of  Interactive  Electronic  Technical  Manuals 


The  technologies  of  computers,  communications,  digital  multimedia  and  distribution 
media  have  provided  new  capabilities  for  publishing  and  distributing  electronic  documents.  We 
now  have  electronic  magazines,  newspapers,  newsletters,  how-to-do-it  manuals,  professional 
conference  proceedings,  and  other  documents  available  through  such  sources  as  CD-ROM  and  the 
Internet.  Both  the  technologies  and  the  tools  for  using  them  are  rapidly  increasing  in  functionality 
and  accessibility.  This,  in  turn,  continues  to  expand  the  types  and  numbers  of  applications  that  are 
developed  as  well  as  changing  the  way  we  perform  many  daily  activities  at  work,  leisure  and  home. 

We  will  discuss  one  particular  class  of  such  documents— Interactive  Electronic  Technical 
Manuals  (lETMs)-that  are  used  as  support  for  job  performance  in  the  military.  In  particular,  these 
manuals  have  been  designed  to  increase  the  effectiveness  of  technicians  who  maintain  equipment 
by  supporting  activities  such  as  troubleshooting,  fault  diagnosis  and  isolation,  and  repairs.  We 
intend  to  illustrate  the  impact  of  such  documents  on  not  only  how  people  access,  navigate,  and  use 
job  documentation  but  also  on  how  they  can  impact  the  job  definition,  all  aspects  of  training,  and 
even  the  organization(s)  that  are  responsible  for  the  job  functions,  training  and  documents 
themselves.  In  the  conclusions  in  Section  6  we  argue  that  the  impact  is  such  that  an  evolution  is 
under  way  and  integration  of  training  and  technical  information  technologies  is  required. 

2.1  Overview  of  lETMs 

To  discuss  the  impact  of  lETMs  on  training  it  is  necessary  to  briefly  describe  what  they  are 
and  why  they  came  to  be.  lETMs  have  several  defining  characteristics  and  this  causes  some 
differences  among  definitions  given  by  various  sources.  We  will  attempt  to  incorporate  the 
important  characteristics  in  the  definition  provided  here.  On  the  one  hand,  lETMs  are  electronic 
documents  with  hypertext,  hypermedia  features.  They  are  viewed  by  means  of  commercially 
available,  or  in  some  cases  government  owned,  viewer  software  on  a  computer  screen  and  can  be 
used  to  access  technical  information  needed  to  troubleshoot  and  help  in  the  repair  of  equipment. 

On  the  other  hand,  lETMs  are  technical  information  databases-consisting  of  text,  graphics, 
tables,  and  potentially  other  objects  of  data— that  also  include  files  that  define  the  documents 
structure  and  content  to  be  viewed.  These  latter  files  identify  Standard  Generalized  Markup 
Language  (SGML)  tag  meanings  related  to  the  objects  of  the  data  base.  SGML  is  a  metalanguage 
(based  on  an  international  standard  ISO  8879),  which  is  extensible  itself,  whose  purpose  is  to 
provide  rules  for  managing,  distributing,  and  publishing  documents.  SGML  is  used  to  define  the 
structure  and  content  of  a  document  but  not  its  appearance  or  presentation.  The  viewing  software 
does  the  latter.  The  result  of  all  of  the  information  in  the  database  is  that  it  is  possible  to  have  a 
highly  interactive  electronic  document  that,  as  we  will  discuss  later,  can  have  the  expertise  built  in 
to  either  solve  equipment  maintenance  problems  or  interact  to  help  the  technician  to  solve  them 
more  quickly  than  with  a  linear,  paper  manual.  The  technician  may  follow  built-in  logic  to  identify 
equipment  symptoms,  possible  problem(s),  and  related  solution(s). 

An  lETM,  then,  is  an  electronic  document  that  is  assembled  for  viewing  from  a  database  of 
technical  information  and  can  interact  with  a  technician  to  experdy  maintain  equipment.  In  actual 
practice,  there  is  wide  variation  on  how  lETMs  look  and  feel,  the  interactivity  that  is  built  in,  and 
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the  way  expertise  is  used  to  guide  the  technician.  This  has  resulted  in  classes  of  lETMs  being 
defined  formally  as  a  way  to  describe  individual  efforts.  Discussion  of  those  classes  and  their 
meaning  will  be  covered  shortly.  Regardless  of  the  class,  it  should  be  recognized  that  the  data  base 
functionality  inherent  in  an  lETM  will  potentially  impact  training. 

2.2  Need  For  lETMs 

It  was  not  only  the  promise  of  technology  that  caused  many  people  in  the  military  to 
investigate  the  possibilities  of  electronic  information  technology  for  replacement  of  paper 
technical  manuals.  The  problems  that  are  inherent  in  the  paper  documentation  system  itself-- 
including  the  life  cycle  of  preparation,  maintenance,  updating,  applicability  to  the  intended  using 
population,  and  logistical  support  for  access— prompted  the  initiatives  toward  electronic  dociunents 
(Rainey,  Fuller,  &  Jorgensen,  1989;  Link,  Von  Holle,  &  Mason,  1987).  One  reason  for  the  problems 
is  that  the  shear  volume  of  manuals  required  to  operate  and  maintain  the  equipment  of  today  is 
immense  and  growing  with  the  increasing  sophistication  of  the  systems.  This  causes  problems  in 
confined  spaces  such  as  on  tanks  and  ships.  The  AEGIS  cruiser,  for  example,  is  said  to  require  65 
tons  of  paper  manuals.  In  addition,  there  were  often  complaints  that  the  paper  manuals  were 
incomplete,  inaccurate,  out  of  date,  poorly  organized  and  difficult  to  use.  Other  factors  contributing 
to  the  problems  were  related  to  the  users  of  the  manuals.  There  were  discrepancies  between  the 
different  reading  levels  of  the  people  using  the  paper  manuals  and  the  single  reading  level  to  which 
the  manuals  were  written.  In  addition,  drops  in  reenlistment  rates  can  mean  continued  use  of  the 
technical  information  by  less  experienced  workers. 

One  might  say  that  with  all  of  those  problems  why  not  fix  the  paper  technical  manual 
system.  Attempts  were  made  to  do  just  that.  However,  part  of  the  problem  is  the  volume  of  the 
paper  needed  to  document  the  information  required.  It  is  not  the  people,  techniques  or 
organizations  that  are  at  fault.  The  volume  of  paper  and  the  logistics  of  its  development  and  support 
were  often  unmanageable.  New  efforts  to  develop  paper  job  aids  which  were  more  complete  and 
better  organized  for  differing  levels  of  users  simply  resulted  in  much  bigger  volumes  of  paper 
(Thomas  &  Clay,  1988).  At  the  same  time,  there  has  been  a  serious  inflation  of  costs  for  paper 
documents  accompanied  by  a  decrease  in  budgets  to  do  everything  necessary,  including 
documentation. 

To  judge  the  viability  of  computer-based  electronic  documents  as  an  alternative,  several 
organizations  in  the  military  conducted  feasibility,  proof  of  concept,  and  demonstration  efforts. 
Although  each  effort  was  done  with  a  limited  set  of  technical  information  on  a  small  scale  of 
participation  and  time,  the  results  were  unequivocally  positive.  For  example,  one  study  (Nugent, 
Sander,  Johnson,  &  Smillie,  1987)  compared  technician  troubleshooting  effectiveness  between  a 
paper  manual  and  a  computer-based  document  that  included  expert  troubleshooting  procedures  and 
guidance  through  diagnostic  events.  The  results  indicated  that  maintenance  technicians  using  the 
computer-based,  electronic  document  as  a  job  aid  completed  troubleshooting  tasks  in  less  than  half 
the  time  it  took  those  with  paper  manuals.  Furthermore,  the  technicians  using  the  electronic  aid 
tested  an  average  of  65%  more  test  points  in  that  reduced  time.  Even  novices  using  the  electronic 
document  aid  could  troubleshoot  12%  faster  than  experienced  technicians  using  a  paper  manual 
(novices  using  a  paper  manual  took  15%  longer  than  the  experts).  Additional  studies  addressed 
maintenance  effectiveness  with  other  equipments,  types  of  technicians,  and  varieties  of  electronic 
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manuals.  Indications  of  high  user  acceptance  and  greater  ease  in  revisability  of  the  documents  were 
also  identified  in  such  studies  (Rainey,  Fuller,  &  Jorgensen,  1989). 

A  shift  in  the  thinking  about  the  role  of  lETMs  as  job  aids  rather  than  simply  electronic 
versions  of  paper  manuals  began  to  surface.  The  small  scale  demonstrations  and  studies  usually 
included  such  techniques  as  coaching  or  other  guidance  for  use  of  the  technical  information  and 
even  expert  troubleshooting  paradigms  to  help  solve  problems.  All  of  these  studies  led  to  wider 
recognition  of  the  gains  that  could  be  obtained  with  electronic  documents,  used  as  job  aids,  for 
maintenance  of  equipment.  As  part  of  this  same  evolution  the  services  formed  joint  working  groups 
to  determine  common  concerns  and  solutions.  Through  Department  of  Defense  Tii- Service  efforts, 
standard  specifications  for  military  wide  development  and  application  of  lETMs  came  about  (MIL- 
M-87268,  1992;  MIL-D-87269,  1992;  MIL-Q-87270,  1992).  There  are  a  large  number  of  lETM 
developments  underway.  Some  are  entering  completion  stages  and  will  soon  be  available  for  use 
on  the  job.  Some  are  available  now.  A  few  are  being  used  in  formal  training  programs  and  many 
more  will  soon  be  entering  that  arena. 

2.3  lETM  Technology 

ETMs  are  possible  because  of  technological  advances  over  the  past  few  years  in  several 
areas.  Some  of  the  same  technology  that  underlies  lETMs  can  also  provide  the  technological  basis 
for  integrating  lETMs  and  training.  Therefore,  it  is  of  value  to  briefly  discuss  some  relevant  aspects 
of  lETM  technology.  Recently,  a  briefing  by  personnel  from  the  Air  Force  Armstrong  Laboratory 
(Masquelier,  1995)  described  the  technological  advancement  for  the  Air  Force  Integrated 
Maintenance  Information  System  (IMIS)  across  three  phases  of  development.  These  are: 

•  Phase  I:  Electronic  Manuals  (1982-1987). 

•  Phase  II:  Interactive  Diagnostics  (1986-1992). 

•  Phase  IE:  Integrated  Information  (1988-1993). 

An  early  report  on  the  state  of  automated  technical  information,  such  as  lETMs,  by 
researchers  at  the  Navy’s  David  Taylor  Research  Center  (Rainey  et  al.,  1989)  described  the 
following  advances: 

•  Advent  of  automated  work  stations  for  creating  and  merging  text  and  graphics. 

•  Greater  information  storage  density  and  retrieval  access. 

•  Increased  capability  of  small  computers. 

•  Development  of  low-cost,  high  resolution  laser  printers. 

•  Improvement  in  software  for  information  scanning. 

•  Improved  display  systems  technology. 

•  Better  database  management  system  software  and  technology. 
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•  Extensive  improvement  in  computer-aided  design  and  engineering. 

•  Advances  in  networking  technology. 

However,  of  all  of  the  technological  advances  underlying  the  interactive  electronic 
document  the  most  dramatic  may  be  the  advent  of  SGML.  Much  of  the  early  work  in  the  military 
on  electronic  technical  manuals  took  place  without  the  power  of  SGML.  It  was  not  until  1986  that 
the  ISO  standards  for  SGML  were  published  (ISO/IEC  8879).  Charles  Goldfarb,  a  lawyer  who 
worked  for  IBM,  invented  a  way  for  lawyers  to  research  a  subject  from  large  text  bases,  edit  and 
add  to  it,  and  thereby  publish  a  document  for  legal  use.  He  developed  SGML  to  fill  a  void  in  the 
word  processing,  publishing  and  retrieval  systems  that  were  available.  None  of  them,  even  in 
combination,  could  do  what  was  needed  and  if  they  came  close  they  could  not  talk  to  each  other 
because  of  proprietary  file  formats  and  inconsistent  treatment  of  text  and  graphics  structures. 
Goldfarb  (Hanlon,  1995a)  realized  that  traditional  publishing  uses  formatting  to  convey  structure 
to  the  reader.  For  example,  to  indicate  a  new  chapter  a  break  occurs  in  pagination  and  a  title  is 
prominently  displayed  at  the  top  of  the  page.  Goldfarb  developed  a  tool  for  describing  the 
underlying  structure  of  the  document  (headings,  abstracts,  sections,  etc.)  regardless  of  the 
formatting  style  to  be  used. 

SGML  fulfills  a  goal  to  produce  a  single  set  of  rules  for  describing  the  structure  and 
managing  the  content  of  a  digital  document.  It  is  an  open  standard  and  a  shared  technology  so  it  is 
not  owned  by  anyone  and  is  available  to  all.  It  is  also  free  from  hardware  and  software  proprietary 
considerations  and,  in  theory,  the  changes  in  hardware  and  software  technology.  A  major  reason 
often  touted  for  incorporation  into  CALS  (Commerce  at  Light  Speed,  formerly  called  Continuous 
Acquisition  and  Life  Cycle  Support)  conventions  is  that  SGML  allows  the  data  to  oudive  the  life 
of  hardware  and  software  available.  The  SGML  rules,  in  the  form  of  a  metalanguage,  provide  a 
way  to  extend  SGML  features  and  we  can  expect  it  to  evolve  with  experience.  The  Hypertext 
Markup  Language  (HTML),  used  to  define  material  on  the  World  Wide  Web,  is  an  example  of  how 
experience  will  change  SGML.  HTML  is  a  specific  subset  of  SGML  rules  used  to  define  WWW 
document  types.  HTML  has  gone  through  several  versions,  some  of  which  bend  current  SGML 
rules,  that  have  been  enhanced  to  meet  project  needs.  Eventually  SGML  may  be  extended  officially 
to  meet  WWW  type  documents. 

Beyond  these  advantages,  SGML  provides  a  new  paradigm  for  working  with  digital 
information.  A  key  aspect  of  the  paradigm  is  that  SGML  is  not  concerned  with  the  appearance  or 
presentation  of  a  document.  An  SGML  document  is  an  information  container,  not  the  screen  or 
hard  copy  output.  A  document  may  be  thought  to  consist  of  three  layers:  structure,  content,  and 
style  (Arbortext,  1995).  SGML  separates  these  three  layers  and  provides  ways  to  directly  define 
the  relationship  between  structure  and  content.  To  do  this  an  SGML  document  can  be  thought  to 
have  three  parts  which,  taken  together,  define  for  a  given  type  of  document  its  structure,  the  content 
of  the  document,  and  the  relationship  of  structure  and  content.  First,  there  must  be  a  declaration;  a 
text  file  that  contains  system  specific  information  such  as  character  sets.  Second,  there  must  be  a 
declaration  of  the  document  type  and  incorporation  of  the  Document  Type  Definitions  (DTD)  that 
are  to  be  used.  The  DTD  is  a  set  of  rules  stated  in  SGML  to  define  the  structure  of  a  document  and 
a  list  of  all  of  the  permissible  elements  and  entities  that  can  be  included  in  the  document.  Elements 
and  entities  are  the  units  of  digital  information  that  are  to  be  included  in  the  document.  Third,  the 
document  instance  consists  of  the  actual  text  with  markups  called  SGML  tags  that  are  defined  in 
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the  DTD.  The  document  instance  is,  therefore,  directly  related  to  the  DTD.  There  are  no  non-text 
objects  in  the  instance  file.  All  Graphics,  for  example,  are  external  to  the  instance  and  referred  to 
as  external  entities. 

The  SGML  tags,  defined  in  the  DTD  and  incorporated  in  the  document  instance  to  define 
the  relationship  between  structure  and  content,  are  the  equivalent  of  penciled  markups  that  are 
often  used  on  the  printed  page  when  we  want  to  identify  characteristics  of  the  document  such  as 
bolding,  position  on  the  page,  or  heading  and  paragraph  types.  However,  the  SGML  markups  do 
much  more.  When  a  tag  is  used  to  identify  a  document  element  such  as  a  unit  of  text  information 
it  can  also  have  attributes  that  add  intelligence  to  the  document,  improve  navigation,  and  trigger 
actions  to  be  taken  automatically.  This  increases  the  user  productivity  by  decreasing  time  to  access 
information  and  providing  a  smart  interface  between  the  user  and  the  document. 

2.4  lETM  Classification  System 

Over  the  past  few  years,  as  the  lETM  specifications  and  standards  were  drafted  and 
promulgated,  it  became  necessary  to  consider  that  not  all  lETMs  can  be  created  equal.  For 
example,  there  will  be  differences  related  to  whether  there  is  a  large  amount  of  legacy 
documentation,  perhaps  in  hard  copy  form  only,  versus  a  new  weapon  system  acquisition  that  can 
allow  authoring  directly  for  screen  interaction  in  logical  information  chunks  versus  whole  linear 
page  displays.  There  will  most  certainly  be  differences  in  budgets  and  time  that  will  effect  how 
much  can  be  done  to  provide  intelligent  documents  which  are  easy  and  functionally  efficient  to  use. 

Given  that  there  are  and  will  be  differences,  the  groups  working  on  lETMs  drafted 
definitions  of  classes  of  electronic  technical  manuals.  This  too  has  been,  appropriately,  an 
evolutionary  process.  The  definitions  have  changed  and  probably  can  be  expected  to  continue  to 
change.  At  one  time,  in  fact,  the  Navy  had  one  classification  system  and  the  Army  and  Air  Force 
at  least  one  other  (Alligood,  1994)  More  recently,  the  lETM  Tri-Service  Working  Group  has 
accepted  the  definitions  shown  in  Table  1.  This  information  (dated  September  2, 1994)  was  taken 
from  the  World  Wide  Web  home  page  of  the  Naval  Surface  Warfare  Center,  Carderock  Division 
(http://navysgml.dt.navy.mil/classes.html).  This  latest  set  of  definitions  differs  in  several  important 
ways  from  a  previous  definition  of  classes  (dated  January  24,  1994).  We  will  briefly  discuss  the 
differences  and  characterize  the  current  definitions  in  terms  of  their  relevance  to  training.  The 
characteristics  of  the  classes  and  the  differences  among  the  classes  are  important  to  the  integration 
of  lETMs  and  training  in  several  ways  that  will  be  expanded  on  later. 
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TVi-Service  Definitions  for  Classes  of  Electronic  Technical  Manuals 
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Content  specific  help  available  •  User-selectable  cross-references  in  session 

and  indices 

•  Content  specific  help  available 


As  seen  in  Table  1,  there  are  three  categories  of  characteristics  used  to  define  lETMs- 
Display,  Data  Format,  and  Functionality.  The  earlier  version  also  had  three  categories— Display, 
Data  Format,  and  Preparation.  There  was  no  specific  reference  to  functionality,  although  such 
items  were  included  to  some  extent  under  Data  Format.  However,  many  of  the  items  currently 
under  the  Functionality  category  in  Table  1  did  not  appear  anywhere  in  the  earlier  version.  These 
differences,  along  the  dimension  of  functionality  from  the  view  point  of  the  end  user  are,  in  fact, 
often  what  distinguishes  one  class  from  another.  One  aspect  of  functionahty  is  how  the  end  user 
will  access,  navigate,  and  interact  with  the  lETM  on-line.  For  example,  we  can  see  increasingly 
greater  use  of  hypertext  and  hypermedia  from  classes  1  through  3  (references  to  hot  spots)  and  the 
addition  of  dialog  driven  interactions  beginning  in  class  3.  Another  aspect  of  functionality  is  how 
much  intelligence  and  expertise  the  system  will  have  that  aids  the  performance  of  the  user.  For 
example,  classes  3-5  provide  logical  next  and  back  functions  for  navigation  and  class  5  is  partly 
defined  by  an  expert  system  that  can  aid  in  implementing  next  navigation  by  taking  into  account 
the  session  data  such  as  troubleshooting  results.  These  functionality  characteristics  potentially 
have  profound  impacts  on  job  definition  and  training  requirements  because  they  may  very  well 
have  major  impacts  on  what  the  technician  has  to  know  and  be  able  to  do.  The  lETMs  have  taken 
on  dimensions  of  electronic  performance  support  systems.  The  importance  of  these  characteristics 
and  the  implications  for  training  will  be  further  discussed  in  Section  4. 

The  Data  Format  category  of  lETM  characteristics  is  of  interest  not  only  for  defining 
classes  but  for  its  training  development  implications.  A  review  across  the  five  classes  along  this 
dimension  reveals  that  the  data  bases  change  in  various  ways,  becoming  increasingly  more  useful 
and  integrated  for  functionality.  Class  1 ,  because  it  starts  out  as  a  legacy  hard  copy  manual,  is  a 
raster  scanned,  page  turning,  electronic  technical  manual  with  only  an  intelligent  index  for 
additional  user  access.  It  has  only  a  bitmap  database  with  indexing  and  header  file  data.  Beginning 
with  class  2,  the  text  data  is  all  digital  with  increasingly  more  useful  and  effective  data  structures, 
use  of  SGML  tagging  to  hierarchically  structure  the  text  data  and  associate  it  to  other  media,  and 
integration  with  other  application  data  both  at  the  database  and  user  level.  This  integration  with 
other  data  is  an  important  principle  that  we  will  discuss  in  Section  4.  In  particular  we  will  argue 
that  common  definitions  of  data  items  from  logistics,  technical  information  and  training  are 
necessary  for  integration  of  lETMs  with  training.  Further,  it  is  likely  that  a  common  database  of 
such  elements  can  serve  the  goal  of  efficiently  training  and  supporting  the  performance  of 
technicians  and  operators  even  better. 

As  a  final  note,  it  should  be  said  that  many  of  the  lETMs  we  have  seen  or  discussed  with 
developers  do  not  readily  fit  into  one  category  or  another.  It  is  common  for  them  to  have  some 
aspects  from  one  class  level  and  some  from  another.  These  multiclass  lETMs  are  not  faulty  in  any 
way.  Rather,  it  is  the  nature  of  lETM  designs,  needs,  budgets  and  developmental  tools  that  seem  to 
dictate  their  characteristics.  Given  the  pressure  to  state  that  our  lETM  is  a  class  X,  we  expect  to  see 
the  definitions  of  classifications  continue  to  evolve  and  become  more  useful  to  define  lETMs  by 
design,  interface  and  functionality  related  to  the  user. 

There  seems  to  be  universal  agreement  that  classes  4  and  5  are  much  more  expensive  to 
develop  than  class  3.  As  reported  by  Hanlon  (1995a),  classes  4  and  5  embed  much  more 
intelligence  into  the  database  (e.g.  hyperlinks,  rendition  information,  context-sensitive  filtering). 
Hanlon  reported  that  93%  of  the  F-16  lETM  developed  by  Lockheed  was  mark-up  as  opposed  to 
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the  actual  text  to  be  displayed  on  the  screen.  This,  too  is  of  relevance  to  the  training  community 
because  the  increased  job  performance  support  that  lETMs  can  have  will  probably  only  be  seen  in 
new,  major,  and  complex  system  acquisitions  with  larger  budgets.  Training  analyses,  including 
media  selection,  will  need  to  take  class  and  category  differences  into  account  in  the  very  near 
future. 


2,5  BETM  Development  Process 

Development  of  lETMs  requires  software  tools  and  processes.  \^ewing  the  lETMs  takes 
still  different  software  tools  called  viewers  or  view  packages.  While  all  lETMs  comply  with  the 
military  standards,  the  process  and  tools  for  development  can  be  considerably  different.  As  we 
surveyed  techniques  and  software  used  to  support  the  authoring  of  lETMs,  it  became  evident  that 
there  were  several  implications  and  issues  for  training  that  we  will  address  later  in  Section  4. 
Specifically; 

•  Should  training  personnel  be  incorporated  into  the  lETM  authoring  process,  and  if  so 
when  and  how? 

•  Can  and  should  training  personnel  incorporate  lETM  processes  and  tools  for  develop¬ 
ment  of  training  materials? 

The  lETM  development  process  varies  from  application  to  application.  Generally  speaking 
there  are  two  major  categories:  (1)  lETMs  developed  for  existing,  mature  systems,  and  (2)  lETMs 
freshly  authored  for  new  systems  and  equipment. 

2.5.1  Mature  System  lETM  Development 

The  following  paragraphs  describe  two  processes  that  have  been  used  to  develop  lETMs 
for  mature  systems  with  legacy  data  in  both  paper  and  electronic  formats. 

Navy  Gas  Ibrbine  lETMs.  Alligood  (1994)  states  that  under  the  direction  of  NAVSEAs 
(Naval  Sea  System  Command)  Technical  Data  Division  (SEA  04TD),  the  following  organizations 
participated  in  the  development  of  the  Gas  Turbine  (GS)  lETM  database:  (1)  the  prime  contractor, 
Lockheed/Martin  (formerly  Martin  Marietta,  Ocean  Systems  Division),  (2)  the  Gas  Turbine 
Systems  In-Service  Engineering  Activity  (GS  ISEA),  Naval  Surface  Warfare  Center,  Carderock 
Division,  Philadelphia  Detachment  (NAVSSES),  and  (3)  New  Technologies  Division  personnel 
and  GS  “C”  school  instructional  staff  at  SSC  GLAKES.  The  GS  lETM  development  process 
followed  the  approach  Lockheed/Martin  used  in  developing  the  Navy  AN/BSY-2  Combat  Systems 
lETM.  The  process,  as  illustrated  in  Figure  1 ,  takes  advantage  of  best  commercial  practices  for  data 
conversion  and  CALS  compliant  tagging  in  SGML  and  best  commercial  practices  for  data 
translation,  integration,  hyperlinking,  and  display. 
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Best  Commercial  Practices 


CALS  Compliance 


Best  Commercial  Practices 


Figure  1.  The  gas  turbine  lETM  development  process. 


Over  110,000  pages  of  technical  information/documentation  were  converted  and  tagged 
during  the  development  of  the  GS  lETM.  NAVSEA  developed  the  Document  Type  Definitions 
(DTD).  The  DTD  consists  of  a  set  of  SGML  tags  (developed  in  compliance  with  MIL-M-28001B, 
August  1993)  that  were  used  to  tag  data  to  the  paragraph  level  as  the  smallest  object  of  data 
converted.  Diagrams,  figures,  and  tables  were  also  SGML  object  tagged.  The  converted  data  was 
uploaded  into  an  object  oriented  common  lETM  database  using  relational  database  management 
software.  The  common  database  consisting  of  two  files— one  file  contains  all  of  the  text  and 
graphics  and  the  object  ID  file  contains  all  of  the  SGML  tags. 

To  create  lETM  functionality,  view  packages  for  general  knowledge,  troubleshooting, 
maintenance,  and  operations  were  developed  by  subject  matter  experts  (SMEs).  Each  view 
package  is  a  script  that  sets  the  flow  from  one  object,  such  as  a  paragraph,  graphic  or  table,  to 
another  that  the  SMEs  consider  the  best  approach  to  accomplish  a  procedure  or  task,  such  as 
troubleshooting  a  specific  piece  of  equipment.  The  SMEs,  consisting  of  personnel  from  SSC 
GLARES  (New  Technologies  Division  and  GS  “C”  school  instructors)  and  NAVSSES  (GS  ISEA), 
were  trained  in  view  package  authoring  techniques  by  Lockheed/Martin  staff.  Development  of  the 
view  packages  was  not  difficult,  but  it  was  a  labor  intensive  effort  over  a  period  of  several  months. 

The  view  packages  were  authored  by  linking,  in  script  format,  the  object  file  IDs  (SGML 
tags)  and  applying  content  data  model  (CDM)  SGML  tags  (defined  in  MIL-M-87269,  November, 
1992).  View  package  scripts  contained  commands  to  link  and/or  launch  to  other  applications  such 
as  video,  audio,  animations,  and  other  documentation  on  compact  disk.  The  scripted  information 
was  uploaded  via  an  ASCII  text  editor  to  the  common  lETM  database  to  provide  positive  linkage 
of  data  in  native  SGML  format.  Then  Lockheed/Martin  processed  the  database  through  a  translator 
for  presentation  on  commercial-off-the-shelf  (COTS)  hypertext/hyperlink  software  (InfoAccess 


GUIDE).  The  lETM  database  was  mastered  on  CD-ROM  for  dissemination  to  the  fleet  and  the 
schoolhouse  to  be  displayed  on  COTS  personal  computer  hardware. 

A  Memorandum  of  Agreement  between  NAVSEA  and  SSC  GLARES  gave  NAVSEA  the 
responsibility  for  developing  the  electronically  enhanced  Instructor  Guides  (IGs)  and  Trainee 
Guides  (TGs)  for  inclusion  in  the  lETM  database.  In  addition  to  the  view  package  development 
effort,  personnel  from  the  New  Technologies  division  and  the  GSE/GSM  instructional  staff  were 
also  rewriting  the  conventional  paper-based  curricula  for  inclusion  in  the  lETM  database.  Rather 
than  revising  the  courses  in  Word  Perfect  and  then  converting  them  over  to  AIM  (Authoring 
Instructional  Materials),  the  Navy’s  current  system  for  authoring  PPP-based  (Personnel 
Performance  Profiles)  instruction,  the  revised  IGs  and  TGs  were  authored  using  AIM.  The  AIM 
databases  were  provided  to  Lockheed/Martin  to  extract  and  convert  the  training  materials  to  SGML 
format  for  inclusion  in  the  lETM  database.  The  IGs  and  TGs  have  the  capability  to  search  for  and 
launch  for  classroom  presentation  to  associated  multimedia  such  as  video,  audio,  and  graphics  that 
are  not  in  the  lETM  database. 

Logistics  Research  and  Development  (LOG  R&D).  One  early  system  used  to  convert 
paper  documents  to  raster  scanned  digital  documents  was  the  Navy’s  Automated  Document 
Conversion  System  (ADCS).  Several  of  those  interviewed,  across  the  services,  expressed 
displeasure  with  the  use  of  ADCS  and  its  final  output.  What  is  most  desirable  is  to  convert  to  a  fully 
digital  format  that  can  be  put  into  an  SGML  database  compliant  with  DETM  specifications.  The 
conversion  of  legacy  data  from  paper  formats  is  an  expensive  process  costing  from  $35-$200  per 
page,  as  estimated  by  the  Naval  Surface  Warfare  Center  (NSWC),  Carderock. 

To  eliminate  or  reduce  those  costs  NSWC  initiated  the  LOG  R&D  program,  funded  by 
NAVSEA,  to  develop  functional  scanning  capabihties  for  conversion  of  paper  based  legacy 
manuals  to  digital  format  compliant  with  the  BETM  specifications.  NSWC  is  looking  for  a  more 
cost-effective  way  to  re-author  legacy  data  from  paper-based  traditional  technical  manuals.  It  is 
envisioned  that  alternatives  for  the  conversion  of  legacy  data  will  require  the  development  of  both 
recognition  and  decomposition  techniques  that  can  analyze  and  identify  technical  information 
needed  to  populate  an  lETM  SGML  database. 

There  are  two  aspects  to  the  problem.  One  aspect  is  to  convert  text,  tables,  references, 
procedural  data,  warnings,  cautions,  and  notes.  The  other  is  to  convert  graphics.  NSWC  has  three 
contractors  tasked  to  develop  systems  for  demonstrations  and  for  evaluation.  Martin  Marietta, 
Orlando,  is  working  on  an  integrated  system  for  both  text  and  graphics.  SAIC,  Tucson,  is  working 
on  a  text  conversion  system.  Mantech  International,  Fairmont,  is  working  on  a  graphics  conversion 
system.  All  have  demonstrated  initial  capabilities.  The  next  phase  of  the  program  will  be  complete 
late  this  year  and  demonstrate  more  complex  format  conversions  to  non-redundant  databases  and 
include  projected  cost  data  for  production. 

2.5.2  Authoring  BETMs  for  New  Systems 

The  process  of  authoring  lETMs  for  new  systems  can  use  tools  specifically  designed  to 
develop  lETMs  in  classes  3,  4  and  5.  Based  on  the  results  of  the  survey,  most  of  these  newly 
authored  lETMs  will  be  at  class  4  or  5  levels.  Since  many  of  the  software  tools  specifically 
designed  for  lETM  authoring  are  under  development,  testing  or  in  early  stages  of  use,  they  are 
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subject  to  change  as  more  experience  is  gained.  Likewise,  general  purpose  COTS  authoring 
systems  may  also  become  more  functionally  related  to  lETM  authoring  as  more  experience  is 
gained. 

We  found  five  lETM  specific  authoring  systems.  Four  of  these  have  some  aspect  owned  by 
the  government.  The  Army,  Navy,  and  Air  Force  have  all  funded  development  of  authoring 
capabilities  as  part  of  lETM  project  development.  The  lETM  specific  authoring  systems,  as 
summarized  from  product  literature  and  interviews,  are  discussed  below. 

AMIDDS.  The  Aviation  Maintenance  Integrated  Diagnostic  Demonstration  System 
(AMIDDS)  is,  like  the  Air  Force  F-16  Integrated  Maintenance  Information  System  (IMIS)  concept 
described  below,  intended  to  provide  an  all  encompassing  and  integrated  support  system  for 
maintenance  of  Navy  aircraft  (Bare  &  White,  1995).  AMIDDS  is  a  Naval  Air  Systems  Command 
R&D  funded  effort  that  includes  support  for  pilot  and  aircrew  debrief  for  fault  identification, 
maintenance  control,  configuration  management,  parts  fife  tracking,  lETMs  and  3M  data 
collection.  AMIDDS  includes  integration  with  expert  system  diagnostics.  lETM  authoring  and 
presentation  software  packages  have  been  developed  for  AMIDDS  by  McDonnell  Douglas 
Aircraft  Company.  It  is,  therefore,  government  owned.  The  authoring  system  has  been  used  to 
develop  lETMs  for  the  F/A-18  program  at  the  class  5  level.  These  prototype  lETMs,  along  with 
other  prototype  components  of  AMIDDS,  are  now  undergoing  field  testing  at  El  Toro  Marine 
Corps  Air  Station.  If  the  testing  is  successful  the  F/A  18  program  will  continue  use  of  the  tools 
resulting  from  the  R&D. 

The  authoring  system  is  an  on-line,  interactive,  set  of  integrated  tools  that  can  be  used  by  a 
technical  writer  to  specifically  develop  lETM  databases  and  documents  at  levels  4  and  5.  Thus,  the 
authoring  process  is  specifically  supported  to  develop  and  declare  elements  and  entities  such  as 
smaller,  logical,  reusable  text  and  graphics  information' without  the  writer  needing  to  know  SGML. 
While  the  output  of  the  database  is  an  SGML  format  for  deliverable  purposes  the  final  database  is 
object-oriented  and  relational  so  that  integration  with  other  such  databases  is  feasible.  The 
authoring  software  runs  on  work  stations  under  an  X  Window  operating  environment.  The 
presentation  software  runs  on  the  same  hardware  and  software  environment  but  is  also  designed  to 
run  on  Portable  Electronic  Display  Devices  (PEDDs)  which  are  ruggedized  and  light-weight  for 
field,  flight-line  usage. 

AIMSS.  The  Advanced  Integrated  Maintenance  Support  System  (AIMSS)  was  developed 
by  Hughes  Technical  Services  Company  to  develop  and  present  class  4  lETMs  conforming  to  the 
lETM  MIL  specifications.  It  is  not  clear  if  any  of  the  system  is  government  owned.  Hughes  offers 
a  COTS  version  for  sale  and,  according  to  Hughes  literature,  the  Army  has  endorsed  AIMSS  as  an 
acceptable  COTS  authoring  environment.  The  system  was  originally  delivered  to  support  two 
Navy  systems:  the  Advanced  Large  Screen  Display  and  a  digital  color  video  generator  unit 
(Hughes,  1995).  Hughes  is  currently  under  contract  to  support  a  Navy  undersea  surveillance 
system  and  a  digital  switching  system  being  integrated  into  the  Canadian  Air  Traffic  Control 
System.  The  AEGIS  lETM  Working  Group  selected  AIMSS  as  the  authoring  system  by  for  all 
AEGIS  cruisers  and  destroyers. 

AIMSS  runs  under  a  Windows-based  environment.  The  authoring  and  run  time  software  is 
integrated  so  it  is  possible  to  immediately  see  how  the  lETM  will  appear  to  the  user.  The  authoring 
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user  interface  is  graphical  and  object-based.  Authors  construct  a  hierarchical  database  structure  by 
using  a  series  of  folders  that  represent  tasks  and  descriptions.  Complex  interactive  procedures  are 
created  by  placing  circles  on  the  screen,  interconnecting  the  circles  and  filling  the  circles  with  text 
and  graphics.  Data  elements  are  sharable  since  they  are  defined  as  objects’  and  can  be  used  many 
times  or  combined  to  define  new  elements.  A  change  in  one  object  is,  therefore,  reflected 
throughout  an  lETM.  AIMSS  provides  point  and  click  hyperlinking  for  the  author;  including  text, 
graphics,  audio,  video,  tables  and  parts  information  objects.  SGML  import  and  export  utilities  are 
included  with  the  authoring  tool  set.  Hughes  provides  full  technical  and  training  support  for 
AIMSS. 

F-16  IMIS.  The  Air  Force  F-16  Integrated  Maintenance  Information  System  (IMIS)  is  a 
system  that  integrates  the  maintenance  data  and  events  through  the  entire  cycle  beginning  with  an 
automated  pilot  debrief  and  going  through  such  events  as  pilot  fault  reporting,  forms  generation, 
both  organizational  and  flight  line  maintenance,  and  post  maintenance  activities  such  as  updating 
database  records  and  closing  out  work  orders  (English  &  Keener,  1995).  The  F-16  IMIS  includes 
an  authoring  system  for  lETMs.  Development  of  the  authoring  system  by  Lockheed  (Ft.  Worth) 
began  in  1989  and  has  been  used  for  the  early  IMIS  field  testing  by  Armstrong  Laboratory  as  well 
as  for  over  9000  maintenance  tasks  on  the  F-16  program.  Tasking  to  Lockheed  calls  for  the  delivery 
of  the  authoring  software  to  the  CALS  program.  Plans  for  further  development  include  merging  the 
F-16  and  F-22  authoring  to  a  single  system,  adding  parts  data  capability,  providing  more  data 
management  tools,  and  including  authoring  features  such  as  audio/video  support. 

The  authoring  system  is  capable  of  producing  a  class  5  lETM  and  the  F-22  lETM  is  a  class 
5  document.  The  rationale  for  making  it  a  class  5  apparently  began  with  an  Air  Force  Inspector 
General  report  that  criticized  the  capability  to  manage  the  maintenance  of  the  F-16  program  across 
four  different  versions  and  with  foreign  military  sales  to  16  countries.  A  class  5  was  set  as  the  goal, 
in  part,  to  integrate  all  data  in  one  data  base  and  be  able  to  draw  from  the  single,  manageable  data 
base  for  all  applications  and  documents.  The  F-16  IMIS  is  currently  being  extensively  tested  at 
several  sites.  However,  there  are  no  current  applications  in  formal  training  programs.  Although 
there  has  been  discussion  of  incorporating  features,  such  audio  and  video,  that  could  be  used  by  the 
training  community,  there  are  no  immediate  plans  to  do  so. 

JIMIS.  The  Joint  STARS  (Surveillance  Target  Attack  Radar  System)  is  also  being 
developed  based  on  the  IMIS  concept.  It  is  being  used  by  Grumman  Aircraft  for  the  Air  Force  Joint 
STARS  program  and  by  the  Boeing-Bell  Helicopter  team  developing  the  Navy  V-22.  The 
authoring  system  for  lETMs  was  developed  by  Grumman  under  Government  funding  and 
therefore  the  Government  owns  the  process  and  parts  of  the  software.  However  both  the  authoring 
system  and  presentation  mode  also  incorporate  COTS.  The  authoring  system  for  both  programs 
incorporate  the  Arbortext  SGML  Publisher.  The  presentation  mode  can  use  a  variety  of  COTS 
viewers.  The  Joint  STARS  lETMs  will  be  Class  4.  They  will  be  ready  in  November/  December, 
1995  for  training  aircraft  differences  to  5  and  7  level  maintenance  technicians  who  will  support  the 
Operational  Test  and  Evaluation  (OT&E)  of  the  aircraft.  The  V-22  lETMs  wiU  also  be  Class  4. 

IADS.  The  U.S.  Army  Interactive  Authoring  and  Display  System  (IADS)  has  been 
developed  as  a  Missile  Command  CALS  initiative.  It  is  Army  owned  and  supported  through  the 
Publications  Service  Branch  of  the  Integrated  Material  Management  Center  at  Redstone  Arsenal. 
IADS  is  compliant  with  lETM  MIL  standards  and  utilizes  SGML  to  manipulate  text  with  CALS 
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compliant  graphics  to  display  isometric  and  engineering  drawings  both  within  text  and  in  separate 
windows.  Both  vector  and  raster  graphics  are  supported  (as  per  MIL-D-28003  CGM  and  MIL-R- 
28002  Raster  Type  I). 

The  software  runs  in  a  Microsoft  Windows  environment  and  requires  a  minimum 
configuration  of  an  80386  with  4  megabytes  of  RAM,  although  more  complex  hypertext 
applications  need  greater  computer  resources.  There  are  two  separate  packages  under  IADS— one 
for  Author  and  one  for  Reader.  The  Reader  software  is  designed  to  display  the  lETM  to  the  end 
user.  The  user  has  the  ability  to  view  and  navigate  documents  to  find  information  needed. 
Navigation  includes  menus,  button  commands,  hot  spots  for  hypertext  links  using  mouse  or 
keyboard  inputs,  and  features  such  as  Next,  Previous,  Back,  and  Search.  Reader  tools  also  include 
the  capability  to  define  private  and  public  notes,  bookmarks,  reports  and  to  print  portions  of  the 
lETM  or  user  defined  text.  Images  displayed  on  the  screen  can  be  manipulated  by  the  user  also 
through  such  features  as  Zoom  and  Image  View.  On-line  help  is  provided. 

The  IADS  Author  package  provides  tools,  with  the  same  features  that  are  in  Reader,  to 
create  and  test  lETM  hypertext  documents  with  SGML  tags  for  elements,  entities  and  attributes. 
Because  a  flat  file  structure  is  used,  the  lETMs  produced  with  IADS  are  inherently  defined  as  class 
3  although  some  other  features  such  as  display  of  small  logical  units  of  information  are  possible 
and  there  may  be  a  mix  of  class  4  and  5  features  that  can  be  added.  Authoring  functions  for  file  and 
frame  properties  allow  fine  tuning  document  file  and  frame  tag  attributes  and  a  Custom  Button 
Editor  and  Control  Panel  Editor  allow  adding,  deleting,  and  modifying  panel  button  characteristics. 
A  style  sheet  package  provides  aids  in  formatting  the  documents  on-screen  appearance.  IADS 
Author  also  includes  a  validating  parser  and  hypertext  link  verifying  program 

The  interesting  aspect  of  IADS  is  that  it  is  being  used  not  only  to  author  lETMs  but  that 
Army  instructional  staff  have  been  trained  to  use  it  as  well.  Field  testing  of  the  Patriot  missile 
lETM  developed  with  IADS  was  completed  at  Ft.  Bliss  in  March,  1995.  The  Integrated  Material 
Management  Center  at  Redstone  Arsenal  has  just  completed  training  of  105  educational  specialists 
and  instructors  on  the  use  of  the  EETMs  and  IADS.  These  personnel  will  be  implementing  the 
lETMs  in  the  classrooms  at  Ft.  Bliss.  They  will  be  able  to  develop  lesson  plans  with  links  into  the 
lETM  through  IADS.  The  lesson  plans  will  be  duplicated  on  student  stations  for  student  automated 
access  to  lETM  material  as  dictated  by  the  lesson  plan.  Only  instructor  stations  will  have  full 
instructor  functions  for  the  lesson  plan. 

It  is  also  possible  for  the  lesson  plans  to  be  built,  through  IADS,  to  launch  other  external 
software  such  as  Asymetrics  Toolbook  applications,  which  is  in  use  now.  Thus,  instructors  may 
incorporate  additional  materials  such  as  graphics,  animations,  and  other  instructional  aids  just  as 
they  would  normally  but  in  a  electronic  classroom  setting.  Discussions  with  another  Army  office, 
has  indicated  that  they  are  working  on  development  of  common  content  tagging  for  use  by 
instructors.  That  will  support  instructor  use  of  IADS  to  link  into  lETMs  from  the  Lesson  Plans 
developed  at  schoolhouses. 
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Section  Three 

Current  lETM  and  Other  SGML  Based  Implementations 


3.0  Current  lETM  and  Other  SGML  Based  Implementations 


The  survey  of  lETMs  covered  military  applications  across  the  Navy,  Marine  Corps,  Army 
and  Air  Force.  In  addition,  we  searched  for  applications  outside  of  the  military  that  were  close  to 
the  characteristics  of  lETMs  by  virtue  of  being  electronic  documents  that  aided  the  completion  of 
a  specific  task  or  immediate  acquisition  of  new  knowledge  that  could  be  used  in  doing  a  task. 

Although  we  initially  considered  looking  at  hypertext,  hypermedia  documents  generally 
without  regard  to  whether  they  were  SGML  based,  we  found  that  the  SGML  applications  were 
much  more  related  to  lETM  technology  and  applications.  Therefore,  we  focused  on  SGML 
applications  in  reviewing  projects  outside  of  the  military.  The  only  exceptions  are  the  research 
projects  on  instructional  hypertextAiypermedia  design  presented  in  Section  5.  These  projects  are 
totally  centered  on  electronic  documents  and  databases  for  learning  systems  as  opposed  to  job 
performance  support. 

The  applications  we  reviewed  are  organized  into  the  following  subsections.  The  lETM 
efforts  in  the  military  will  be  presented  first.  The  lETM  projects  in  other  government  agencies  and 
industry  will  be  discussed  next  Finally,  the  applications  found  in  higher  education  will  be 
described.  In  each  section,  we  will  attempt  to  characterize  the  diverse  efforts  in  order  to  draw  out 
implications  for  the  future. 

3.1  Military  lETMs 

CALS  (Continuous  Acquisition  and  Life  Cycle  Support)  was  initiated  in  1985  by  DoD  as 
a  strategy  for  integrating  logistic  support  processes.  In  1988,  DoD  mandated  compliance  with 
CALS  specifications.  lETM  specifications,  considered  a  p^  of  CALS,  were  published  in  late 
1992.  For  the  Navy,  electronic  forms  of  technical  information  and  documentation  have  been 
directed  by  the  Chief  of  Naval  Operations  in  the  CALS  Strategic  Plan  (OPNAVINST  4120.5,  Jan 
1994).  Other  military  branches,  such  as  for  the  Army,  through  its  Materiel  Command,  have  also 
directed  replacement  of  paper  technical  manuals  with  DETMs. 

Table  2  summarizes  many  of  the  DoD  lETM  applications  that  were  identified  during  the 
survey.  There  are  undoubtedly  more,  but  these  seem  to  the  major  ones  commonly  known  in  the 
community. 

While  all  of  the  lETMs  shown  in  Table  2  comply  with  the  lETM  military  specifications, 
there  are  substantial  differences  in  them.  For  example,  no  two  necessarily  look  alike.  In  addition, 
even  though  they  all  comply  with  SGML  rules  they  cannot,  at  the  SGML  database  level,  talk  to 
each  other  through  the  same  software  without  some  steps  for  conversion.  There  are  a  wide  variety 
of  lETM  authoring  software  tools  and  techniques,  presentation  software  packages,  features  for 
functionality  and  integration  with  other  applications  or  databases,  and  levels  of  expertise  built  in 
for  job  performance  support. 
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Table  2 


Some  Military  lETMs  Currently  in  the  Field  or  Under  Development 
BRANCH  AND  PROGRAM  RELATED  INFORMATION 

Navy/Marine  Corps,  V-22  In  Progress,  Class  4  being  developed  with  JIMIS  authoring  system  by 

BeU-Boeing.  Expected  to  be  fielded  for  first  flight  of  plane  #7  in 
December,  1996 

Navy/Marine  Corps,  APG-65  Radar  System  In  preparation  as  a  Class  5  with  C-h-  software  to  integrate  graphics 

used  in  F-18  and  AV-8B  Aircraft  and  animation  produced  with  COTS.  Intended  to  have  full  integration 

of  lETMs  with  Computer  Based  Training. 

Navy/Marine  Corps,  F/A-18  AMIDD  R&D  effort  in  Field  Testing  at  El  Toro,  Class  4/5  being  developed 

with  AMIDD  authoring  system  by  McDonnell  Douglas.  If  R&D  is 
successful  the  F/A-18  program  will  pick  up  the  tools  developed  for 
further  application. 

Fielded  and  in  School  at  Great  Lakes.  Utilizes  electronic  classroom 
technology.  Developed  by  Martin  Marietta  with  InfoAccess  Guide 
Author.  EBTDynatext  is  viewing  software.  IGs  and  TGs  are  authored 
in  WordPerfect  and  not  integrated  with  the  lETMs,  Class  3 

Fielded  and  in  School  at  Great  Lakes.  Utilizes  electronic  classroom 
technology.  Developed  by  Martin  Marietta  with  InfoAccess  Guide 
Author.  IGs  and  TGs  are  authored  in  AIM  and  integrated  with  the 
lETMs.  Class  3 

Navy,  Radio  Communication  System  (RCS)  Class  3  fielded  in  1991  and  still  in  place.  Used  in  training  at  AEGIS 

for  AEGIS  Training  Center. 

Navy,  Fire  Control  Transmitter  for  AEGIS  S  Class  4  developed  under  NSWC  PHD  with  Hughes  AIMSS  authoring 

system.  Under  evaluation  now. 

Navy,  AN/BSY-2  Seawolf  Class  Submarines  Combat  system  Class  3  lETM  developed  with  InfoAccess  Guide 

Author.  Developed  by  Martin  Marietta.  EBT  Dynatext  is  viewing 
software.  lETMs  to  be  incorporated  into  training  at  Submarine 
School,  Groton.  Will  be  fielded  in  1996.  IGs  and  TGs  are  integrated 
and  authored  with  the  lETMs 

Navy,  HME  &  Systems  for  Seawolf  Hull,  Mechanical,  Electrical  &  systems  level  lETMs  being  developed 

at  Class  2  &  3  levels  primarily  by  Newport  News  &  Electric  Boat  with 
InfoAccess  Guide  Author.  EBT  Dynatext  is  viewing  software.  lETMs 
to  be  incorporated  into  training  at  Submarine  School,  Groton. 

Navy,  AN/BQG-5  for  Trident  Submarines  Passive  sonar  receiving  set  currently  fielded  onboard  USS  Augusta 

SSN710  as  Class  3  EETM.  Developed  by  Martin  Marietta  with 
InfoAccess  Guide  Author.  EBT  Dynatext  is  viewing  software.  Being 
used  in  an  electronic  class.  IGs  and  TGs  are  integrated  and  authored 
with  the  EETMs. 

Sonar  system  Class  3  to  be  developed  at  the  Fleet  Technical  Support 
Center,  Dam  Neck.  Not  under  way  at  time  of  interview. 

R&D  program  to  build  a  functional  scanner  of  paper  legacy  manuals. 
See  the  description  in  this  report  on  lETM  authoring  of  legacy  data 
lETMs 


Navy,  SQQ-89  for  SPA  29G 
Navy,  LOG  R&D 


Navy,  Gas  Turbine,  Electrical  and 
Mechanical 


Navy,  Mine  Counter  Measures 
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Table  2  (Continued) 


BRANCH  AND  PROGRAM 

RELATED  INFORMATION 

Navy,  Advanced  Tomahawk 

Expected  in  late  1996  at  the  Fleet  Combat  Training  Center,  Dam 

Neck. 

Navy,  MK  75/76 

To  be  used  at  the  Fleet  Combat  Training  Center,  Dam  Neck 

Navy,  Close-in  Weapon  System 

To  be  used  at  the  Reet  Combat  Training  Center,  Dam  Neck 

Army,  Paladin  Howitzer 

In  Progress.  Aug  1997 

Army,  Towed  Howitzer 

Fielded 

Anny,  Palletized  Load  System  (PLS)  Vehicle 

Fielded  and  used  in  training  at  Odnance  Center  &  School  (OC&S) 

Army,  Highly  Mobile  Multi-Wheel  Vehicle 
(HMMWV) 

In  progress.  May  1996 

Army,  Armored  Personnel  Carrier 

In  pa-ogress.  December  1995 

Army,  M915/916  Trucks 

Fielded  and  used  in  training  at  Ordnance  Center  &  School  (OC&S) 

Army,  All  Source  Analysis  System 

Fielded 

Army,  Armored  Gun  System  (AGS) 

In  progress.  4th  Qtr  FY  1996. 

Army,  Joint  Tactical  Unmanned  Vehicle 

In  progress.  4th  Qtr  FY  1996. 

Army,  Team  Mate 

Fielded 

Army,  Trail  Blazer 

Fielded 

Army,  Heavy  Equipment  Transporter 

Fielded  and  used  in  training  at  Ordnance  Center  &  School  {OC&S) 

Army,  Explosive  Ordinance  Disposal 

Fielded  but  just  received  at  joint  service  school  (at  NSWC,  Indian 
Head)  which  is  reviewing  how  to  integrate  into  training. 

Army,  Apache  Longbow  Attack  Helicopter 

In  progress.  1997.  Class  5  because  it  is  being  refurbished  and 
upgraded  for  long  life. 

Army,  Patriot  Missile  System 

Fielded  and  now  in  schools  at  Ft.  Bliss.  Class  3 

Army,  Avenger  Missile  System 

Fielded  and  now  in  schools  at  Ft.  Bliss.  Class  3 

Army,  Blackhawk  Helicopter 

Field  Testing 

Air  Force,  Joint  STARS  Aircraft 

Ready  for  Fielding  Nov/Dec  1995  as  Class  4  and  will  be  used  in 
training  then  for  maintenance  and  some  mission  crew  operations. 

Air  Force,  F-16  Aircraft 

Class  5,  in  Field  but  not  in  schools 

Air  Force,  F-22  Aircraft 

In  progress.  Class  5 

Air  Force,  B-2  Aircraft 

In  progress.  Class  5 

All  Source  Analysis  System 

Fielded. 
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As  shown  in  the  table,  most  of  the  lETMs  are  under  development.  Only  ten  lETMs  are  now 
in  the  schools.  No  Air  Force  lETMs  have  been  incorporated  into  formal  training  to  date  although 
they  will  be  shortly  in  several  instances.  The  five  Navy  school  lETM  applications  are  the  electrical 
and  mechanical  Gas  Turbine  Technician  courses  at  Service  School  Command,  Great  Lakes,  the 
Mine  Counter  Measures  course  (just  begun  this  summer  with  lETMs)  also  at  Great  Lakes,  the  AN/ 
BQG-5  sonar  set  course  at  the  Trident  Training  Facility,  and  the  radio  control  system  incorporated 
at  the  AEGIS  Training  Center.  The  two  of  the  five  Army  lETMs  in  schools  are  for  the  Patriot 
Missile  and  Avenger  Missile  at  Ft.  Bliss.  The  three  other  Array  lETM  based  training  programs  are 
at  the  Ordnance  Center  &  School.  These  ten  applications  are  all  reported  to  be  in  electronic 
classrooms,  although  the  components  in  the  classrooms  vary  considerably.  (Refer  to  Section  4.0 
for  a  discussion  of  lETMs  in  electronic  classrooms).  The  five  Navy  school  lETMs  have,  or  will 
have,  the  Instructor  and  Trainee  Guides  for  formal  curricula  integrated  with  the  lETMs  and 
distributed  on  the  same  CD-ROMs.  All  of  the  schools  have  integrated  the  lETMs  into  hands-on 
labs.  The  OC&S  school,  for  example,  has  integrated  a  simulator  that  can  be  set  yp  as  a  truck  with 
faults  entered  by  the  instructor.  From  the  students  viewpoint  the  contact  data  test  set  he/she  is  using 
is  the  real  thing  connected  to  a  real  truck. 

Although  there  were  some  instances  of  using  lETMs  in  on-demand,  just-in-time  training 
modes,  these  usually  seemed  to  be  impromptu  and  informal  situations  at  a  unit  level  rather  than 
regular  and  planned  in  conjunction  with  training  commands  or  other  activities  charged  with  broad- 
based  training.  The  AN/BQG-5  lETM  was  reported  to  have  been  used  for  on-demand  training  to 
various  groups  such  as  shipyard  installation  personnel  and  government  engineers.  The  integrated 
curriculum  in  that  lETM  was  developed  using  an  Audience  Tailored  Instruction  development 
process  that  allows  any  training  objective  to  be  mapped  into  an  on-demand  training  topic,  course, 
or  overview. 

We  found  no  instances  of  complete  virtual  classrooms  applications  such  as  networking 
lETM-based  training  via  satellite  to  ships  or  remote  sites.  However,  several  organizations  have 
experimented  with  elements  of  this.  The  Navy  has  experimented  with  use  of  two  way  audio/  video 
to  show  an  instructor,  using  an  lETM,  while  talking  to  a  technician  on  board  a  ship.  An  Army 
Training  and  Doctrine  command  (TRADOC)  funded  initiative  is  now  underway  to  begin 
teletraining  that  integrates  lETMs.  The  Army,  at  the  Redstone  Arsenal  Office  of  the  Test, 
Measurement,  and  Diagnostic  Equipment  Program  Manager,  has  a  planned  contract  effort  to  study 
use  of  the  World  Wide  Web  for  lETMs  access  for  truck  and  tank  maintenance.  While  these  are  not 
specifically  training  oriented,  they  are  perhaps  precursors  to  looking  at  the  technology  basis  for 
virtual  access  to  lETMs  and  training.  A  Web  site  is  now  in  operation  by  the  Army  to  support 
operations  at  the  Ordnance  Center  &  School.  That  Web  site  will  be  turned  on  for  all  Army  access 
in  November,  1995. 

The  lETMs  shown  in  Table  2  vary  considerably  in  terms  of  features.  They  range  from  a  few 
class  2  applications  to  classes  4  and  5.  There  are  several  class  1  lETMs/ETMs  but  they  were  not 
included  since  they  are  more  archival  than  active  and  not  likely  to  be  of  importance  in  training.  The 
class  2  and  3  lETMs  are  all  for  existing  systems  and  equipment  belonging  to  mature  programs 
where  existing  manuals  were  used  as  the  basis  for  developing  the  lETMs.  The  new  programs,  such 
as  the  Navy  V-22  and  the  Air  Force  Joint  STARS,  are  funded  well  enough  to  be  developed  at  a  class 
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4  level.  Other  programs,  such  as  the  Air  Force  F-16  and  F-22  programs,  are  funded  adequately  to 
be  developed  at  a  class  5  level. 

We  were  not  able  to  identify  consistent  and  specific  criteria  for  decision  making  on  what 
class  a  given  lETM  should  be  across  the  services.  However,  two  other  criteria  stand  out,  beyond 
the  criteria  of  funding  availability  and  whether  technical  manuals  are  already  in  existence.  One  of 
these  is  the  desirability  of  incorporating  expert  system  technology  into  features  such  as  diagnosis 
and  coaching  of  technicians  through  troubleshooting.  This  is  an  especially  relevant  criteria  where 
complex  equipment  failures  are  time  critical  or  personnel  may  not  be  available  with  necessary 
skills  for  these  activities.  A  second  criteria  is  die  desirability  of  integration  of  databases  and 
applications  in  a  total  maintenance  system.  For  the  Army,  for  example,  if  the  equipment  has  a  bus 
system  that  wdl  support  integration  with  a  total  maintenance  diagnostic  system  the  lETM  will  be 
developed  at  a  level  5. 

The  Army  decisions  about  their  desired  lETM  features  illustrate  how  these  criteria  come 
into  play  and  how  those  decisions  can  impact  job  definitions  and  the  training  required  for  the  job. 
These  decisions  also  illustrate  how  an  lETM  can  be  integrated  with  other  applications.  The  Army 
has  decided  to  combine  ETM/IETMs  with  computer-aided  diagnostics  and  test  procedures  to  form 
a  total  maintenance  system.  In  part,  this  is  desirable  because  units  such  as  trucks  and  tanks  will 
often  be  forward  of  an  organized  maintenance  capability  and/or  moving  too  quickly  under  adverse 
conditions  to  bring  the  equipment  into  a  shop.  Personnel  on  the  vehicles  cannot  be  trained  for  all 
maintenance  situations  without  smart  aids.  Furthermore,  the  drawdown  in  the  Army  has  resulted 
in  fewer  maintainers  for  more  complex  systems.  Often  consolidation  of  MOSs  has  not  included 
more  formal  schooling,  which  means  there  is  a  need  for  more  job  performance  support,  especially 
in  the  field. 

A  family  of  equipment,  included  as  part  of  standardization  for  automatic  test  systems, 
includes  three  versions  of  Contact  Test  Sets  (CTSs)  designated  the  AN/PSM-80(V).  These  three 
versions  of  CTSs  are  capable  of  not  only  connecting  to  equipment  for  test,  measurement  and 
diagnosis  but  also  of  delivering  lETMs.  All  three  are  based  on  Intel  i486  processor  configurations. 
Version  1  is  designed  primarily  to  host  delivery  of  ETMs/IETMs.  Version  2  is  also  a  self  contained 
test  and  diagnostic  system  capable  of  connecting  to  a  variety  of  vehicles  through  its  interface 
cables.  Version  3  is  a  self  contained  system  primarily  designed  to  be  a  generic  automated  weapon 
system  tester.  It  is  being  used  across  a  diverse  group  of  systems  including  the  Avenger  missile 
system.  Paladin  Howitzer,  and  Abrams  tank. 

The  treatment  of  lETMs  as  part  of  a  total  diagnostic  system  has  been  described  by  SAIC, 
a  contractor  who  has  developed  many  of  the  vehicle  lETMs  and  diagnostic  systems,  as  having  the 
following  capabilities: 

•  The  host  computer  connects  to  the  vehicle  to  be  worked  on  and  the  lETM  senses  built- 
in  test  failure  information 

•  If  such  information  is  present  the  lETM  directs  the  user  to  the  correct  repair  proce¬ 
dure. 
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•  The  host  computer  also  connects  to  troubleshooting  instruments  and  the  lETM  takes 
measures. 

•  Based  on  the  measures  the  lETM  directs  the  user  to  the  correct  maintenance  proce¬ 
dure. 

•  The  lETM  automatically  links  to  the  correct  entry  in  the  RPSTL  (Repair  Parts  and 
Special  Tools  List)  which  is  normally  another  manual  and  would  involve  a  task  for 
another  repair  person. 

•  The  BETM  automatically  inserts  repair  information  into  appropriate  blocks  of  mainte¬ 
nance  forms. 

•  The  lETM  assists  in  transmitting  maintenance  information  to  the  Unit  Level  Logistics 
System. 

Since  each  CTS  costs  from  $15,0000-$20,000,  it  is  expensive  to  outfit  entire  classrooms 
with  them.  However,  the  CTS  design  is  flexible  to  allow  standard  Intel  486  based  desktop 
computers  to  play  the  lETMs  without  the  integrated  diagnostic  maintenance  capability.  The  Patriot 
school,  for  example,  only  has  a  few  CTSs  for  hands-on  training  and  plays  the  lETMs  on  standard 
desktop  PCs  for  the  remainder  of  training. 

The  Air  Force  philosophy  of  integrated  maintenance  illustrates  another  way  the  criteria  to 
design  lETMs  is  used.  The  Air  Force  concept  for  maintenance  is  called  Integrated  Maintenance 
Information  system  (IMIS)  and  is  part  of  the  Air  Force  CALS  initiative.  IMIS  was  started  for  much 
the  same  reasons  as  the  programs  in  the  Navy  and  Army.  There  are  more  complex  systems  to 
maintain  and  more  computerized  maintenance  aids  for  collecting  data  and  diagnosing  problems  are 
needed. 

One  difference,  shared  by  the  Navy/Marine  Corps  and  some  Army  air  maintenance 
organizations,  is  the  need  to  focus  on  aircraft  maintenance.  Therefore,  the  concept  includes  total 
integration  from  the  time  the  aircraft  is  in  flight  and  on-board  diagnostics  are  running,  to  the 
aircrew  debriefs  on  faults,  scheduling  of  maintenance  actions,  status  reporting,  parts  ordering, 
repair  and  troubleshooting.  The  technician  uses  a  portable  maintenance  aid  that  contains  ETMs/ 
lETMs,  diagnostics,  training,  and  data  collection  capability.  Data  is  downloaded  from  the  aircraft 
to  the  portable  device.  The  device/IETM  displays  inspection  procedures  to  the  technician,  presents 
repair  procedures  tailored  to  the  data,  and  performs  operational  checks  to  verify  repairs.  When  the 
maintenance  actions  are  complete  IMIS  transmits  maintenance  data  collection  reports  and  closes 
out  work  orders.  IMIS  also,  conceptually,  contains  supplemental  training  material  such  as 
computer  based  instruction  and  collects  training  data  from  technicians  interacting  with  that  training 
although  we  were  not  able  to  confirm  that  is  currently  being  done  on  any  program. 

A  class  3  lETM  cannot  fulfill  the  IMIS  concept.  The  IMIS  concept  requires  class  4  and  5 
lETMs  because  the  units  of  information  must  be  small  and  logically  tied  to  the  diagnostics  system 
and  other  integrated  aspects  of  IMIS.  Further,  the  data  base  features  and  capabilities  needed  to 
integrate  with  other  application  databases  move  the  lETM  levels  to  class  4  or  5. 
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Classes  4  and  5  are  without  a  doubt  the  lETMs  that  have  the  potential  to  redefine  a  job  and, 
therefore,  the  training  required.  They  may  also  have  the  greatest  potential  for  integrated  training 
elements.  Unfortunately,  they  are  also  much  more  expensive  than  classes  1-3  and  we  are  likely  to 
see  most  lETMs  developed  at  a  class  3  level  or  lower. 

3.2  Other  Government  and  Industry  SGML  Documents 

Arbortext  (1995)  estimates  that  business  in  this  country  produces  92  billion  documents 
every  year  and  that  number  is  growing  substantially.  Managing  and  accessing  that  information  is 
becoming  an  increasingly  more  difficult  problem.  Government  agencies  (non  DoD)  produce 
equivalent  mountains  of  documents  and  share  those  problems.  For  that  reason,  several  government 
agencies  and  industry  groups/consortiums  have  initiated  efforts  to  apply  SGML  technology  to 
electronic  documentation.  This  section  discusses  some  of  these  efforts. 

When  we  started  the  survey  of  industry,  we  were  looking  for  lETM-type  publications, 
whether  SGML  or  not,  that  had  training  and  education  integrated  or  linked  otherwise  in  some  way. 
We  also  expected  to  find  a  number  of  Electronic  Performance  Support  Systems  (EPSS)  with 
documents  similar  to  lETMs  that  were  integrated  with  training.  With  the  exception  of  the  air 
transportation  industry,  we  did  not  find  such  efforts.  Although  there  are  a  number  of  EPSS  in  place 
(Gery,  1991),  they  do  not  have  large  scale  hypertext/hypermedia  Job  documents  as  a  major 
component  integrated  with  training.  Therefore,  we  only  included  SGML  applications  that  might 
lead  us  to  implications  for  integration  of  training  in  the  survey. 

Table  3  summarizes  the  projects  we  found.  Further  information  on  most  of  these  can  be 
obtained  from  an  SGML  Web  site  at  http://www.sil.org/sgml/sgml.http.  The  applications  shown  in 
Table  3  include  the  air  transportation  industry,  several  federal  government  agencies, 
telecommunications  industry,  music  publishing/business  interests,  electronic  document  publishers, 
software  industry  publications,  news  and  newspaper  industry,  groups  seeking  to  advance  access  to 
documents  for  print  disabled  people,  and  an  electronic  library  of  chemical  society  journals.  We 
believe  that  there  are  many  more,  but  these  applications  are  certainly  representative. 

Of  all  the  applications  shown  in  Table  3,  the  first  one,  the  Air  Transportation  Association 
(ATA),  is  the  most  relevant  The  following  description  of  efforts  are  taken  from  separate 
discussions  with  six  Boeing  employees,  each  in  a  different  area  of  responsibility.  We  have  taken 
the  liberty  of  making  a  composite  picture  of  what  might  be  accomplished  although  there  is 
apparently  no  explicit  initiative  to  move  in  this  direction  now. 

Boeing  has  two  initiatives  related  to  electronic  manuals.  Both  of  these  are  expected  to  be 
on-line  around  the  end  of  1995.  One  is  associated  with  development  of  an  ATA  standard  for 
electronic  books  including  maintenance  manuals  and  parts  catalogs.  This  standard  starts  with 
defining  SGML  data  elements,  but  adds  the  feature  of  Structured  Full  Text  Query  Language 
(SFTQL).  The  latter  will  be  incorporated  into  a  special  DTD  to  define  a  query  schema  for  the 
electronic  books.  The  second  initiative  related  to  electronic  manuals  is  to  put  the  technical 
information  in  manuals  in  electronic  form  and  distribute  (download)  them  to  portable  maintenance 
aids  in  the  field.  This  is  the  Integrated  Technical  System  Initiative  (ITS A),  which  is  very  much  like 
the  lETM  concept.  The  computers  will  have  customized  search  criteria  to  guide  troubleshooting, 
defined  through  intelligence  that  is  incorporated  into  the  DTD  and  acted  on  by  the  viewer  software. 
All  COTS  software,  such  as  authoring  software  from  Interleaf  Corporation  and  EBTs  Dynatext,  is 
used  to  accomplish  this. 
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Table  3 


SGML  Projects  in  Other  Government  Agencies  and  Industry 


PROJECT  NAME 

Air  Transportation 
Association  (ATA) 

Boeing  Commercial 
Aircraft  Company 

The  Department  of  Energy 
(DOE)  Office  of  Scientific 
and  Technical 
Information  (OSTI) 

IPI-TCIF  (Information 
Products  Interchange/ 
Telecommunications 
Industry  Forum) 


SMDL  -  Standard  Music 
Description  Language, 
ISO/mCDIS  10743: 
1995 


IRS  (US  Internal  Revenue 
Service) 


BRIEF  DESCRIPTION  OF  PROJECT 

Boeing  has  several  related  initiatives  that  are  intended  to  result  in  the  capability  to 
distribute  electronic  manuals  for  maintenance  of  aircraft  on  the  flight  line  as  well  as 
integrate  with  training  in  the  classroom,  simulators,  and  on-demand,  just-in-time 
settings. 

OSTI’s  activities  include  extensive  use  of  SGML.  "...One  of  the  initiatives,  for  example, 
is  the  Department- wide  effort  to  implement  standards  which  will  allow  for  the  full-text 
electronic  exchange  of  scientific  and  technical  documents.  OSTI  also  serves  as  the 
central  point  for  the  collection  and  sharing  of  scientific  and  technical  information 
management  activities  of  interest  to  DOE  and  its  customers.” 

Extracted  from  the  IPI  description  file,  dated  August  1994  at 
info.bellcorexom/pub/rCIF/ipi_descJxt,  “The  Information  Products  Interchange 
Committee  addresses  one  of  the  largest,  most  widespread  problems  in  the 
telecommunications  industry  -  the  vast  amount  of  paper  documentation  associated  with 
equipment  used  in  the  telecommunications  network.  With  the  ready  availability  of 
terminals,  computers  and  publishing  and  retrieval  systems  for  creating,  viewing  and 
managing  documents  in  electronic  form,  delivery  of  the  documentation  electronically  is 
a  logical  step.” 

“When  the  Telecommunications  Industry  Forum  (TCIF)  precursor  of  the  formal  PI 
Committee  began  meeting,  it  was  quickly  apparent  that  most  companies  had  dissimilar 
platforms,  operating  systems,  work  cultures  and  processes.  No  company  could  afford  to 
abandon  its  investment  in  tools,  and  most  felt  their  operation  was  ^propriate  for  their 
circumstances.  This  meant  that  the  members  of  a  different  committee  needed  to  address 
recognized  standards  and  develop  guidelines  that  could  support  a  diverse  industry  and 
support  the  emerging  information  technologies  rather  than  to  try  to  enforce 
standardization  of  tools.” 

“The  IPI  Committee  selected  Standard  Generalized  Markup  Language  (SGML)  as  the 
standards  for  exchange  of  text.  As  steps  toward  implementing  SGML,  the  committee 
has  developed  an  SGML  Document  Type  Definition  (DTD),  conducted  document 
analysis  for  telecommunications  procedural  documents  and  successfully  completed  trial 
exchanges  of  documents  in  an  SGML  Format.  The  committee  also  works  with  Bellcore 
on  the  development  of  Generic  Requirements  Documents  on  SGML.” 

Extracted  from  [July  1995]  DIS  (FTP.ORNL.GOV/PUB/SGMLAVG8/SMDL/10743: 
“This  International  Standard  defines  an  architecture  for  the  representation  of  music 
information,  either  alone  or  in  conjunction  with  text,  graphics,  or  other  information 
needed  for  publishing  or  business  purposes.  Multimedia  time  sequencing  information  is 
also  supported.  The  architecture  is  known  as  the  “Standard  Music  Description 
Language”,  or  “SMDL”.  //  SMDL  is  a  “Hy-Time”  application,  it  conforms  to 
International  Standard  ISO/IEC  10744  -  Hypermedia  /  Time-based  Structuring 
Language  (“HyTime”).  Specifically,  SMDL  is  a  “derived  architecture”  derived  from 
HyTime  architecture” 

A  large  number  of  IRS  documents  are  available  in  SGML  format  in  the  IRS-SGML 
Library  on  the  FedWorld  Information  Network.  The  SGML  documents  are  complete 
with  DTD.  The  IRS-SGML  Library  had  some  90  documents  as  of  April  11, 1995,  and 
the  list  is  said  to  be  “updated  each  morning.”  The  documents  are  encapsulated  in  self- 
extracting  binary  files  (“.EXE”  MS-DOS  compressed  files). 
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Table  3  (Continued) 


PROJECT  NAME 

Association  of  American 
Publishers  (AAP) 
Electronic  Publishing 
Special  Interest  Group 
(EPSIG) 


Davenport  Group 
[DocBook  DTD] 


CADD:  International 
Committee  on  Accessible 
Document  Design 


UTF  (Universal  Text 
Format)  -  SGML  for  the 
News  Distribution 
Industry 


CAPS:  Communication 
and  Access  to 
Information  for  Persons 
with  Special  Needs 


BRIEF  DESCRIPTION  OF  PROJECT 

EPSIG  was  invited  by  the  AAP  to  carry  out  development  and  promotion  of  the  EPSIG/ 
AAP  “Electronic  Manuscript  Standard”  for  the  preparation,  publication  and  interchange 
of  electronic  manuscripts.  The  most  visible  element  of  the  new  EPSIG’s  charter  is  to  be 
the  “voice”  of  “ISO12083:  1993,”  reporting  on  implementation,  modification,  and 
evolution.  In  (about)  early  1994,  the  sponsorship  of  the  Electronic  Publishing  Special 
Interest  Group  (EPSIG)  was  transferred  from  OCLC  [On-line  Computer  Library  Center] 
to  management  under  three  collaborating  entities.  These  are:  the  Association  of 
American  Publishers  (AAP)  [supplies  direction  for  SGML  requirements  of  the  AAP 
membership],  the  Graphic  Communications  Association  Research  Institute  (GCARI) 
[organizational  management],  and  McAffe  &  McAdam,  Ltd  [technical  direction]. 

The  DocBook  DTD  was  developed  specifically  for  computer  software  documentation, 
that  is,  user  manuals  and  programming  references.  DocBook  maintenance  is  performed 
under  the  aegis  of  the  Davenport  CJroup,  a  discussion  forum  sponsored  by  individuals 
representing  large-scale  producers  and  consumers  of  software  documentation. 
Davenport  meetings  are  held  roughly  quarterly,  and  are  open  to  everyone.  However, 
decisions  about  the  DocBook  DTD  are  made  only  after  approval  by  the  Davenport 
sponsors.  As  of  11  August  1995,  the  Davenport  sponsors  were:  Novell,  Fujitsu  OSSI, 
Digital  Equipment  Corporation,  SunSoft,  Hewlett-Packard,  Arbortext,  SoftOuad, 
Hitachi  Computer  Products,  SCO,  O’Reilly  &  Associates 

SGML  is  being  used  by  various  entities  supporting  IC  ADD  because  the  document 
structure  captured  in  SGML  is  vital  in  the  communication  processes  designed  for 
persons  with  print  disabilities.  ICADD  Statement  of  Purpose:  The  International 
Committee  for  Accessible  Document  Design  (ICADD)  is  dedicated  to  making  printed 
materials  accessible  to  persons  with  print  disabilities.  ICADD  is  an  international 
nonpartisan  consortium  of  representatives  from  industry,  education,  and  the  disabled 
community.  We  believe  that  advancing  computer  based  publishing,  through  adaptive 
computer  technology  for  persons  with  disabilities,  offers  the  potential  to  make  printed 
information  accessible  simultaneously  and  at  no  greater  cost  than  the  able  bodied 
community  enjoys.” 

UTF  (‘Universal  Text  Format”)  is  the  name  of  a  new  standard  being  adopted  by  the 
news  distribution  industry,  and  particularly,  under  the  direction  of  working  committees 
in  the  IPTC  (International  Press  Telecommunications  Council)  and  NAA  (Newspaper 
Association  of  America).  UTF  is  part  of  The  Information  Interchange  Model  (DM) 
standard.  Description  from  a  working  document  on  UTF  by  Dave  Becker  “In  June, 
1992,  a  working  subcommittee  was  established  to  create  an  industry  standard  for  the 
interchange  of  textual  material  between  news  agencies  and  their  clients  (primarily 
newspapers)  that  would  replace  the  current  standard  IPTC  7901  and  ANPA  1312  for¬ 
mats.  The  new  standard  is  called  the  Universal  Text  Format  (UlT).  After  significant 
discussion,  SGML  was  adopted  as  the  encoding  language  for  the  new  standard. 
Members  of  the  working  subcommittee  are  now  attempting  to  finalize  and  prototype  the 
new  standard  in  selected  test  environments.  “ 

The  CAPS  project  has  developed  a  DTD  for  newspapers 
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Table  3  (Continued) 


PROJECT  NAME  BRIEF  DESCRIPTION  OF  PROJECT 

SEC  (Securities  SGML  Markup  EDGAR  filings 

Exchange  Commission) 

EDGAR  Database 

The  OCLC  (On  Line  The  CORE  project  is  an  electronic  library  prototype  that  provides  networked  access  to 
Computer  Center)  CORE  the  fuU  text  and  graphics  content  of  the  American  Chemical  Society  journals  and 
Project  associated  Chemical  Abstracts  Services  indexing  since  1980  (some  250  journal  years  of 

data).  The  database  is  coded  in  SGML  (Standard  Generalized  Markup  Language)  which 
was  translated  from  the  original  typography  codes,  captures  the  structural  richness  of  the 
original  document  and  provides  flexibility  for  indexing,  searching  and  display.  The 
prototype  provides  a  full-scale  laboratory  environment  in  which  to  explore  1  issues  of 
database  structure,  user  interface  capabilities,  and  information  retrieval  1  questions  on  a 
large,  real-world  scholarly  electronic  journal  database.  The  complete  database, 
representing  more  than  6(X),000  pages  of  full  text  and  graphics,  will  be  available  at 
Cornell  University  in  late  1994.  The  major  contributors  of  this  electronic  library  project 
include:  Cornell  University  (Mann  Library);  OCLC;  Bellcore;  American  Chemical 
Society;  Chemical  Abstracts  Services.” 
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While  the  primary  reason  for  moving  to  electronic  manuals  in  the  aircraft  industry  is  the 
desire  to  eliminate  the  cost  and  problems  of  paper  management,  they  have  recognized  that  they 
have  the  opportunity  to  deliver  just-in-time  training  through  the  same  technology.  Another 
initiative  will  incorporate  SGML  based  documents  for  training  on  the  flight  line.  This  effort  is  in 
the  planning  stage  for  a  prototype.  This  training  is  not  intended  to  take  the  place  of  classrooms  and 
simulators,  especially  for  initial  training.  As  pointed  out  by  one  of  the  interviewees,  classroom 
training  focuses  on  principles  and  has  less  details  whereas  the  just-in-time  training  envisioned  for 
the  field  focuses  on  the  details.  To  accomplish  this,  some  people  from  the  training  departments  are 
involved  in  the  electronic  manual  development.  High  level  management  is  committed  to  the 
integration  of  electronic  books  and  training  as  part  of  ITS! 

Other  aspects  of  new  Boeing  training  initiatives  and  planning,  some  of  which  support 
moving  training  from  the  classroom  to  a  just-in-time  setting,  are  development  of  paper  workbooks 
with  bar  code  testing  devices,  computer-based  training  (CBT),  distance  learning,  and  collaborative 
learning.  Most  of  the  current  maintenance  training  is  held  in  a  classroom  setting.  For  example,  the 
Aviation  Industry  CBT  Committee  (AICC)  has  been  looking  at  object-oriented  authoring  and 
studying  what  structure  is  needed  for  a  data  base  to  support  putting  instruction  on  hand  held  devices 
outside  the  instructor  led  classroom  environment.  In  part,  this  initiative  is  also  oriented  toward 
integrating  the  documentation  and  training  worlds.  As  pointed  out  by  another  interviewee  we  are 
changing  the  paradigm  of  what  we  think  of  as  training  and  must  redefine  when  and  what  training 
is  related  to  such  capabilities  as  smart  documents,  on-demand  training  and  just-in-time  training. 

3.3  SGML  Applications  in  Higher  Education 

Most  of  the  applications  found  in  higher  education  relate  less  to  lETMs  per  se  and  more  to 
the  use  of  SGML  and  other  aspects  of  lETM  technology  for  case-based  (Kolodner,  1993; 
Montague,  1994)  military  education  including  complex,  advanced  learning  such  as  weapon  system 
acquisition,  weapon  system  management,  joint  force  planning,  and  military  strategy  and  tactics. 
Case-based  instruction  provides  the  learner  with  a  meaningful,  real  world  context  by  which  to 
anchor  new  knowledge  and  promote  transfer  of  using  that  knowledge  in  new  situations.  Case- 
based  instruction  is  often  thought  of  as  a  classroom-based  technique,  but  it  is  sometimes 
incorporated  into  CBT  as  well.  We  see  the  value  of  converting  case  studies  in  such  areas  to 
structured  SGML  electronic  documents  that  provide  smart  tools  for  the  learner  to  navigate,  search 
and  link  relevant  case  material  as  they  plan,  design  and  solve  problems.  While  such  electronic 
documents  can  be  expected  to  enhance  learning,  they  can  also  be  made  available  outside  the 
classroom  environment  perhaps  through  facilities  at  networked  schools  such  as  the  various  military 
academies  and  post  graduate  schools  and  even  the  World  Wide  Web. 

Most  of  the  SGML  applications  found  in  higher  education  are  designed  to  support  scholarly 
research  efforts  in  the  humanities,  literature,  arts,  and  other  content  domains.  A  great  deal  of  work 
is  being  done  by  university  libraries  as  part  of  movements  to  make  electronic  documents  available 
for  scholarly  research.  Of  special  interest  here  is  the  work  being  done  to  make  the  documents  more 
readily  accessible  via  an  interface  that  is  meaningful  to  the  scholar.  This  meaningful  interface  can 
translate  to  support  for  a  learner  in  a  case-based,  SGML  encoded  learning  environment.  For 
example,  the  LETRS  (Library  Electronic  Text  Resource  Service)  at  Indiana  University  is  designed 
to  support  access  to  large  text  documents  by  humanist  scholars  in  the  way  that  they  tend  to  ask 
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questions.  This  focus  on  the  particular  scholars  mental  model  for  asking  questions  is  an  important 
way  to  approach  design  and  structuring  of  the  documents. 

Another  example  of  user  interface  consideration  is  the  Rossitti  Archive,  at  the  Institute  for 
Advanced  Technology  in  the  Humanities  (lATH).  It  is  a  hypermedia  environment  encoded  with 
SGML  for  studying  the  works  of  a  poet  and  painter.  This  application  illustrates  the  display  of 
SGML  based  hypermedia  with  annotations  and  notes  from  experts.  The  same  technique  could  be 
applied  to  military  case-based  education  and  training.  Software  developed  by  the  lATH  represents 
still  other  aspects  of  the  learner  interface  that  are  relevant  to  military  case  based  learning.  It 
provides  an  SGML-aware  synoptic  tool  that  can  display  multiple  texts  in  linkable  parallel 
windows.  It  will  allow  multilingual  texts  to  be  displayed  simultaneously.  Users  will  be  able  to  link 
the  texts  for  scrolling  and  searching.  This  capability  would  allow  a  learner  to  build  new  displays 
and  files  that  connect  information  related  to  a  case  or  problem  given  in  a  learning  environment. 

Further  information  on  the  applications  shown  in  Table  4  is  available  through  the  SGML 
Web  site  at  http://www.sil.org/sgml/sgml.http. 
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Table  4 


EBgher  Education  Applications  of  SGML  Technology 

Project  name  brief  description  of  project 

Text  Encoding  Initiative  The  TEI  has  developed  an  SGML  encoding  for  a  wide  range  of  document  types  in  the 

(TEI)  domain  of  humanities  computing.  The  Text  Encoding  Initiative  is  an  international 

research  projdbt  sponsored  by  the  Association  for  Computing  in  the  Humanities  (ACH), 
the  Association  for  Literary  and  Linguistic  Computing  (ALLC),  and  the  Association  for 
Computational  Linguistics  (ACL).  Funding  has  been  provided  in  part  by  the  US 
National  Endowment  for  the  Humanities,  Directorate  XEII  of  the  Conunission  of  the 
European  Communities,  the  Andrew  W.  Mellon  Foundation,  and  the  Social  Science  and 
Humanities  Research  Council  of  Canada.  The  TEI  (“P3”)  Guidelines  were  published  in 
May  1994,  after  six  years  of  development  involving  many  hundreds  of  scholars  from 
different  academic  disciplines  worldwide. 

Oxford  Text  Archive  The  Oxford  Text  Archive  has  sponsored  extensive  research  involving  the  use  of  SGML 
(OTA)  in  an  academic  setting.  A  catalog  of  electronic  texts  in  the  Archive  is  available  in  SGML 

format.  OTA  is  also  the  authoritative  FTP  site  for  a  significant  corpus  of  literary  texts 
encoded  in  (TEI)  SGML  by  members  of  the  Oxford  Text  Archive  project  and  by  others.. 

University  of  Michigan  The  Humanities  Text  Initiative  (HTI)  is  a  project  of  the  University  of  Michigan  Libraries, 

the  UM  Press,  and  the  School  of  Library  and  Infomxation  Studies,  with  support  from  the 
College  of  Literature,  Science  &  Arts.  The  HTI  is  responsible  for  creating  and  maintain¬ 
ing  new  textual  collections,  primarily  in  TEI  SGML. 

University  of  Virginia  The  University  of  Virginia  has  pioneered  a  number  of  successful  uses  of  TEI  SGML  in 

Electronic  Text  Center  delivering  on-line  electronic  texts,  including  structured-text  searches.  “The  Electronic 

Text  Center  at  the  University  of  Virginia  combines  an  on-line  archive  of  thousands  of 
SGML-encoded  electronic  texts  with  a  library-based  Center  housing  hardware  and  soft¬ 
ware  suitable  for  the  creation  and  analysis  of  text.  Through  ongoing  training  sessions  and 
support  of  individual  teaching  and  research  projects,  the  Center  is  building  a  diverse  and 
expanding  user  community  locally,  and  providing  a  potential  model  for  similar  enterpris¬ 
es  at  other  institutions,  “(from  the  ETC  home  page) 

Brown  University  STG  “supports  the  development  and  use  of  advanced  information  technology  in  academ- 

Scholarly  Technology  ic  research,  leaching,  and  scholarly  communication.  SGML  is  used  in  several  aspects  of 

Group  (STG)  STG  work,  and  most  visibly  in  the  Women  Writers  Project.  The  Women  Writers  Project 

is  creating  a  full-text  database  of  women’s  writing  in  English  from  the  period  1330-1830. 
Texts  are  encoded  in  TEI  SGML.. 

British  National  Corpus  The  BNC  is  a  large  (100  million  words)  corpus  of  modem  English,  both  spoken  and  writ- 

Project  (BNC)  ten,  produced  by  an  academic/industrial  consortium  lead  by  Oxford  University  Press,  in¬ 

volving  Longman  UK  Ltd,  Chambers/Larousse,  Oxford  University  Computing  Services, 
the  University  of  Lancaster  and  the  British  Library.  Production  of  the  corpus  was  funded 
by  the  conunercial  partners  and  by  the  UK  Government,  under  the  DTI/  SERC  Joint 
Framework  for  Information  Technology,”  The  corpus  is  encoded  according  to  the  TEI 
(Text  Encoding  Initiatives  Guidelines,  using  the  ISO  standard  SGML  to  represent  this 
and  a  variety  of  other  structural  properties  of  texts  (e.g.  headings,  paragraphs,  lists  etc.). 
Full  classification,  contextual  and  bibliographic  information  is  also  included  with  each 
text  in  the  form  of  a  TEI  conferment  header  file.” 
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Table  4  (Continued) 


PROJECT  NAME  BRIEF  DESCRIPTION  OF  PROJECT 

(Library  Electronic  Text  LETRS  provides  on-Une  access  to  SGML  tagged  texts.  LETRS  supports  the  rapid  index- 
Resource  Service)  ing,  searching,  retrieving,  and  navigating  of  large,  structured,  full  text  files  in  ways  rele- 

LETRS  Indiana  vant  to  the  types  of  questions  that  humanists  ask  about  texts.  Texts  are  tagged  in  SGML 

University  (Standard  Generalized  Markup  Language)  and  conform  to  the  TEI  (Text  Encoding  Ini¬ 

tiative)  implementation  of  that  standard. 

Institute  for  Advanced  Several  lATH  projects  use  SGML  encoding  in  the  preparation  of  electronic  scholarly  text 

Technology  in  the  editions.  Software  developed  under  lATH  auspices  has  occasionally  been  released  as 

Humanities  (lATH)  well.  For  example,  in  early  1995,  lATH  announced  Babble:  a  synoptic  text  viewer.  Cur- 

University  of  Virginia  at  rently  under  development,  it  will  be  an  SGML-aware  synoptic  text  tool  that  can  display 
(Charlottesville  multiple  texts  in  linkable  parallel  windows.  It  will  allow  multilingual  texts  to  be  dis¬ 

played  simultaneously.  Users  will  be  able  to  link  the  texts  for  scrolling  and  searching. 

lATH:  Rossetti  Archive  The  Rossetti  Archive  at  lATH  features  the  writings  and  pictures  of  Rossetti  encoded  in 

SGML.  The  Rossetti  Archive  is  a  hypermedia  environment  for  studying  the  works  of  the 
Pre-R^haelite  poet  and  painter  Dante  Gabriel  Rossetti  (1828-1882).  The  archive  is  a 
structured  database  holding  digitized  images  of  Rossetti’s  works  in  their  original  docu¬ 
mentary  forms.  Rossetti’ s  poetical  manuscripts,  early  printed  texts  -  including  proofs  and 
first  editions  -  as  well  as  his  drawings  and  paintings  are  stored  in  the  archive,  in  full  color 
as  needed.  The  materials  are  marked  up  for  electronic  search  and  analysis,  and  they  are 
supplied  with  full  scholarly  annotations  and  notes 

Center  for  Electronic  CETH  is  sponsored  joindy  by  Rutgers,  the  State  University  of  New  Jersey  and  Princeton 

Texts  in  the  Humanities  University.  CETH  acts  as  a  national  focus  for  the  creation,  dissemination  and  use  of  elec- 
(CETH)  tronic  texts  in  the  humanities  with  emphasis  on  scholarly  applications  and  primary  source 

Rutgers  University,  New  materials.  CETH’s  activities  include  an  Inventory  of  Electronic  Texts  in  the  Humanities 
Brunswick,  New  Jersey  and  research  into  methods  of  providing  Internet  access  to  collections  of  SGML-encoded 

material  in  the  humanities.  In  particular,  CETH  has  promoted  the  study  and  use  of  the 
TEI  (SGML)  header  as  a  form  of  authentication  in  cataloging  electronic  texts  by  libraries 
and  archive  centers. 

Berkeley  -  Berkeley  With  assistance  from  Electronic  Book  Technologies!  Educational  Grant  Program  and  the 

Finding  Aid  Project  NEH  (Act  Tide  IIA  Research  and  Demonstradon  Grant.  October  1993  -  September 

University  of  California  1995),  the  project  is  sponsoring  advanced  research  in  SGML  technologies  for  use  by 
at  Berkeley  electronic  libraries.  Finding  aids  are  documents  used  to  describe,  control,  and  provide  ac¬ 

cess  to  collections  of  related  materials.  In  the  hierarchical  strucmre  of  collection  level  in¬ 
formation  access  and  navigation,  finding  aids  reside  between  bibliographic  records  and 
the  primary  source  materials. 


36 


Table  4  (Continued) 


PROJECT  NAME 


BRIEF  DESCRIPTION  OF  PROJECT 


Project  ELSA 
(Electronic  Library  SGML 
Applications) 


Project  ELSA  is  carrying  out  research  into  the  use  of  documents  in  libraries  which  have 
been  marked  up  in  SGML  format  (Standard  Generalized  Markup  Language).  The  project 
will  construct  an  electronic  store  of  documents  which  will  take  the  form  of  a  server  on  a 
netwoik.  Client  computers  will  be  able  to  access  the  material  on  the  server,  download  it 
and  make  it  available  to  librarians  and  end  users  for  use  and  manipulation.  The  project  is 
funded  through  European  Commission  DG  Xin  and  has  three  partners.  Jouve  System 
D’ Information  (France)  the  lead  partner,  will  provide  the  search  engine,  user  interface 
software  and  client  server  software,  Elsevier  Science  (Netherlands)  will  provide  the  doc¬ 
uments  and  De  Montfort  University  will  develop  the  user  interface  and  provide  a  test  bed. 


UCLA  -  InfoUCL A  The  UCLA  information  processing  group  has  been  experimenting  with  SGML,(Standard 

Project  Generalized  Markup  Language)  as  a  way  to  publish  its  data  so  that  it  can  be  accessed  in 

a  variety  of  ways,  one  of  which  is  via  the  International  Committee  for  Accessible  Docu¬ 
ment  Design  (IC  ADD)  document  type  definition,  which  through  the  use  of  enabling  tech¬ 
nologies  makes  electronic  information  accessible  to  people  with  disabilities. 
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Section  Four 


Integration  of  lETMs  with  Training  and  Education 


4.0  Integration  of  lETMs  with  Training  and  Education 


The  focus  in  this  report  now  turns  to  the  training  of  technicians  with  lETMs  as  a  primary 
media  and  content  resource.  Since  technical  manuals,  whether  paper  or  electronic,  usually  form  a 
major  basis  in  training  programs  for  learning  how  to  perform  tasks  and  apply  knowledge,  it  is 
expected  that  introducing  lETMs  into  training  will  change  things. 

4.1  BETM  Ikaining  Settings 

These  changes  will  be  discussed  within  the  context  of  the  traditional  and  electronic 
classroom;  the  movement  of  training  out  of  the  classroom  to  on  demand,  just-in-time  training;  and 
the  reduction  of  training  by  increasing  the  lETMs  role  as  job  aids.  Each  of  these  has  been  a  reaction 
to  the  conceptual  nature  of  lETMs  and  is  now  a  path  being  taken.  Since  they  represent  quite 
different  approaches  it  is  worth  examining  the  rationale  and  potential  of  each.  Regardless  of  the 
approach,  it  can  be  said  with  certainty  that  lETMs  have  changed  the  way  documentation 
organizations  do  their  work  and  will  change  the  way  training  organizations  do  business  as  well. 

4.1.1  lETMs  in  the  Classroom 

As  discussed  previously,  the  early  rationale  for  lETMs  centered  on  improving  the 
documentation  system.  Improving  technician  proficiency  and  thereby  increasing  maintenance 
effectiveness  also  became  a  goal  when  the  lETMs  were  defined  with  interactive,  job  aiding 
features.  It  soon  became  apparent  that  if  these  job  aiding  electronic  documents  could  increase 
maintenance  technician  effectiveness,  even  to  the  point  of  bringing  novices  to  the  level  of  more 
experienced  people,  there  might  be  an  impact  on  the  training  needed.  That  is,  maybe  the  time 
required  to  train  could  be  reduced  thereby  reducing  training  costs  and  increasing  availability  of 
people  to  perform  jobs.  For  example,  a  33%  reduction  in  the  number  of  training  courses  required 
and  a  28%  reduction  in  the  number  of  training  days  was  estimated  for  technicians  of  one  large 
weapon  system  training  program  (Alligood,  1994).  With  those  goals  as  an  incentive,  attention  has 
recently  turned  to  the  technicians  formal  training  programs. 

Much  of  the  training  of  military  technicians  occurs  in  centralized  schoolhouses  where  the 
facilities  and  other  resources,  such  as  instructors  and  laboratory  equipment,  can  be  made  available. 
In  most  cases,  the  instruction  occurs  in  combinations  of  classroom  and  laboratories.  The  traditional 
classrooms  are  typically  instructor-led  with  lectures,  demonstrations,  and  whatever  interaction 
with  students  time  allows.  The  introduction  of  lETMs  means  putting  computers  in  the  classrooms 
and  laboratories.  We  will  discuss  the  planning  and  logistical  requirements  associated  with 
installing  computers  in  the  classrooms  in  Section  4.2.  In  the  remainder  of  this  section,  we  will  focus 
on  the  impact  on  the  instructional  nature  of  the  classroom. 

The  introduction  of  computers  into  the  classrooms  and  laboratories,  for  the  purpose  of 
supporting  instruction  using  lETMs,  also  parallels  the  movement  towards  the  electronic  classroom. 
There  has  been,  over  the  past  few  years,  much  interest  in  the  electronic  classroom  in  the  military, 
business  and  industry  training,  and  in  the  educational  sectors  at  all  levels.  By  electronic  classroom 
we  mean  the  use  of  computers  and  communications  to  support  the  development,  access,  and 
management  of  learning  resources  and  events. 
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During  a  recent  survey  of  electronic  classrooms  (Kribs,  1994),  we  found  intense  activity  in 
the  military  and  some  of  these  were  linked  to  the  coming  of  lETMs  into  the  training  programs. 
Since  the  lETMs  and,  therefore,  computers  would  be  used  in  the  training  program  a  number  of 
questions  naturally  arose.  For  example,  how  can  an  instructor  control  and  monitor  the  activities  at 
each  student  station?  How  can  an  instructor  use  the  lETMs  to  display  specific  electronic 
information  such  as  pictures,  graphs  and  tables,  to  the  class?  How  can  the  lETM  material  be 
augmented  with  instructional  material  such  as  additional  graphics,  animations  and  photos  which 
the  instructors  would  like  to  use?  What  additional  functions  can  the  computers  perform,  such  as 
testing  and  record  keeping,  remediation,  interactive  courseware,  and  collaborative  learning?  These 
questions  are  being  addressed  and  lETM-based  electronic  classrooms  are  gradually  being 
redefined  and  extended  to  meet  the  needs 

Our  survey  of  electronic  classrooms,  along  with  work  by  others,  has  demonstrated  that 
there  is  no  single  definition  of  an  electronic  classroom  that  fits  all  situations.  Furthermore,  we 
believe  that  concepts  of  electronic  classrooms  will  and  should  evolve.  Several  close  observers  of 
electronic  classrooms  in  the  educational  sector  have  recently  commented  on  the  variations  in 
definitions  and  the  possible  transformation  of  educational  practices  and  organizations  as  a  result  of 
technology.  Plater  (1995)  of  Indiana  University  Purdue  University,  for  example,  has  noted  that  the 
term  has  become  such  a  symbol  of  change  in  education  that  it  is  often  used  to  even  describe  pseudo- 
electronic  classrooms  that  consist  only  of  a  large  amount  of  technology,  sufficient  to  emulate  some 
interactivity  and  communication  but  which  lack  the  communication  and  connectivity  processes 
students  and  teachers  really  need.  As  Plater  points  out,  however,  by  any  definition  the  first 
electronic  classroom  in  any  organization  should  be  viewed  as  the  beginning  of  a  process  that  may 
change  the  institution.  He  offers  four  categories  of  change  which  should  be  considered:  (1)  when, 
where,  and  how  people  learn;  (2)  what  and  why  they  learn;  (3)  the  evolving  role  of  the  faculty 
(toward  enabling  students  to  learn  themselves  rather  than  as  transmitters  of  knowledge);  and  (4) 
the  future  of  the  institution  itself. 

Similarly,  Connick  and  Russo  (1995)  of  the  Education  Network  of  Maine  argue  that 
technologies,  such  as  telecommunications,  challenge  the  fundamental  tenets  upon  which  our  entire 
educational  system  has  been  constructed  (p  15).  They  point  to  the  changing  need  to  aggregate 
people  simply  to  communicate  with  them  and  the  movement  toward  learner-centered  learning 
environments.  Whether  one  agrees  fully  with  these  practitioners,  it  is  still  obvious  that  experience 
with  lETM  based  electronic  classrooms  will  suggest  changes  in  the  way  we  do  business.  Virtual 
classrooms  are  a  distinct  future  possibility  for  lETM  based  training  as  the  movement  toward  a  more 
learner  centered  electronic  classroom  progresses. 

To  some  extent,  the  current  mental  model  of  the  classroom  as  an  instructor  led,  on-the- 
podium,  group  learning  setting  is  now  being  simply  transferred  over  to  the  lETM-based  electronic 
classrooms.  We  do  not  argue  against  this  as  a  first  step,  but  suggest  that  changes  in  student  learning 
and  the  instructor  roles  may  be  worth  considering  in  the  next  few  years.  A  recent  Issue  Paper  by 
staff  at  the  Navy’s  Service  School  Command,  Great  Lakes  (Fretz,  1995)  illustrates  one  thought  on 
this  topic  based  on  the  lessons  learned  from  one  of  the  first  lETM  based  electronic  classrooms.  A 
computer  classroom  can  be  more  than  an  enhancement  of  the  classic  group  paced  instruction  (GPI). 
Computers  are  best  used  to  present  self-paced  instruction  with  the  instructor  becoming  a  moderator 
and  mentor.  Our  lETM  [-based  classroom]  followed  the  GPI  mold  due  to  curriculum  control 
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concerns,  the  cost  of  developing  15  weeks  of  quality  self-paced  instructional  modules  (ICW),  and 
the  primary  objective  of  taming  the  paper  tiger  first  and  breaking  the  paradigm  second  (p.  1).  It  is 
suggested  that  not  only  can  the  instructors  role  change  but  so  can  the  students.  The  lETM-based 
electronic  classroom  can  be  stmctured  for  students  to  take  more  responsibility  for  themselves  and 
with  each  other. 

A  recent  development  that  should  be  of  interest  to  all  who  are  involved  with  electronic 
classrooms  is  going  on  at  the  Navy’s  Great  Lakes  Service  School  Command.  The  Training 
Integration  Management  Software  (TIMS,  formerly  called  the  Annotation  Manager)  is  being 
developed  by  the  contractor  System  Engineering  Associates  under  the  Classroom  Automation 
System  research  program  at  the  Navy  Personnel  Research  and  Development  Center.  TIMS  is 
designed  to  provide  instructors  with  the  ability  to  personalize  the  curricula,  as  well  as  integrate  and 
operate  the  variety  of  windows-based  applications  in  the  classroom.  For  example,  TIMs  provides 
the  facilities  to  control  classroom  multimedia  and  lETM  presentations.  Instructors  can  alsouse 
TIMS  functions  to  personalize  a  standard  instructor  guides  with  text  and  other  digital  media  forms. 
TIMS  allows  linking  lETMs  to  the  standard  instructor  guide  so  that  the  instructor  can  instantly 
launch  the  lETM  viewer  at  the  place  desired  in  the  curriculum.  TIMS  uses  the  standard  Windows 
communication  facilities,  such  as  dynamic  data  exchange  (DDE)  that  allows  applications  to 
exchange  data.  It  is  written  in  Microsoft  Visual  Basic  with  some  third  party  DLLs  (dynamic-link 
libraries). 

4.1.2  lETMs  in  On-Demand  and  Just-In-Time  IVaining 

The  lETM-based  electronic  classroom,  according  to  estimates  by  several  Navy  sources,  is 
anticipated  to  result  in  fewer  courses  and  less  training  time.  This  new  form  of  the  classroom  will 
also  result  in  changes  in  the  roles  of  instructors  and  students.  In  addition,  it  can  be  expected  that 
the  definition  of  the  classroom  will  continue  to  change  as  its  functionality  is  enhanced  by  the  new 
technology  available.  The  virtual  classroom  may  be  one  of  those  ways  it  changes.  The  networked 
virtual  classroom,  whether  for  physically  co-located  groups  or  physically  separated  individuals, 
offers  the  promise  of  distributing  the  instruction  to  more  people  with  less  instructor  resources. 

Several  military  agencies  responsible  for  the  implementation  of  lETMs  and  training  are 
studying  the  option  to  move  more  training  out  of  the  classroom  by  supporting  performance  in  the 
field  with  smart  lETMs  and  computer  based  training  (CBT)  that  can  be  accessed  as  needed. 
Although  no  organization  we  surveyed  was  considering  total  eUmination  of  classrooms  and 
associated  laboratories,  especially  for  initial  training  of  apprentices,  there  was  a  definite  inclination 
to  reduce  the  formal,  centralized  school  time.  This  initiative  revolves  around  the  concept  of  just- 
in-time  training  with  a  performance  support  system.  Just-in-time  training  seems  especially  to  be  a 
focus  for  the  military  and  industry  areas  of  aircraft  maintenance.  Both  the  Navy  and  Air  Force  are 
planning  for  lETMs  and  CBT  as  a  means  of  providing  technicians  performance  support  on  the 
flight  line  and  for  in-service  training.  Several  of  the  major  aircraft  manufacturers  we  surveyed  are 
also  developing  SGML  based  electronic  documents,  with  intelligence,  to  aid  their  customers 
technicians  on  the  flight  line.  These  electronic  documents  are  also  part  of  a  total  concept  of  a 
performance  support  system  that  includes  computer  based  training. 

There  are  three  points  we  want  to  mention  briefly  in  association  with  just-in-tirae  training 
performance  support  systems.  First,  part  of  this  concept  is  the  idea  of  an  integrated  data  base  of 
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technical  information  that  can  be  used  for  both  documentation  and  training  application  software. 
That  is,  the  integration  of  technical  information  needed  for  both  maintenance  and  training  can 
reduce  redundancy  in  both  development  and  configuration  management  efforts,  allow  more  timely 
updates  of  both  the  documentation  and  the  training  in  concert,  and  permit  one  agent  to  manage  the 
data  base.  Second,  the  possibility  exists  to  generate  the  CBT,  or  at  least  a  first  iteration  of  it,  from 
the  technical  information  of  the  data  base.  Both  of  these  will  be  discussed  further.  Third,  the 
concept  of  such  a  performance  support  system  raises  interesting  questions  about  the  assessment  of 
effectiveness.  How  do  we  know  which  component  of  the  performance  support  system  is  working 
or  not  working  when  data  is  gathered  to  assess  performance  itself?  We  suggest  that  assessment 
evaluation  needs  to  be  part  of  the  total  system. 

4.1.3  lETMs  as  Expert  Job  Aids 

Several  of  the  organizations  interviewed  expressed  strong  preference  for  moving  the 
capability  to  support  the  technician  more  via  the  lETMs  and  providing  whatever  training  is  needed 
in  the  field.  The  Army,  for  example,  is  faced  with  the  prospect  of  a  fast  moving  set  of  vehicles 
racing  across  land  well  beyond  reach  of  expert  maintainers  and  their  logistical  resources.  The  more 
intelligence  that  can  be  put  into  an  lETM  the  better  the  operators  of  the  equipment  can  be  supported 
to  fix  what  breaks.  In  fact,  this  has  been  extended  in  several  cases  to  provide  higher  levels  of 
automation  in  the  troubleshooting  and  repair  process. 

As  described  in  our  review  of  lETMs  in  the  military,  the  Army  is  developing  many  lETMs 
that  are  embedded  into  a  diagnostic  system.  That  is,  the  lETM  computer  connects  to  the  equipment 
being  worked  on  and  the  lETM  senses  built-in-test  information  to  direct  the  technician  to  the 
correct  repair  procedure.  The  lETM  computer  also  connects  to  troubleshooting  instruments  to 
allow  sensing  readings  and  provide  direction  of  the  use  of  the  instruments  as  part  of  the  repair 
process.  When  the  lETM  has  identified  the  component  to  be  replaced,  it  displays  and  guides  the 
technician  through  the  correct  procedure.  The  lETM  then  automatically  links  to  the  correct  entry 
in  the  repair  parts  and  special  tools  list  (usually  another  manual  the  technician  must  use).  Finally, 
the  lETM  automates  the  filling  out  of  maintenance  records  and  sets  up  the  basis  for  transmitting 
that  information  to  the  logistics  system  later.  This  is  certainly  a  redefinition  of  jobs  and  training 
requirements  as  a  result  of  lETM  technology. 

However,  we  note  that  not  all  lETMs  will  have  such  a  capability.  The  cost  of  developing 
class  4  and  5  lETMs,  versus  class  3,  may  be  a  stumbling  block.  The  economics  of  lETMs  is  beyond 
the  scope  of  this  report,  but  we  hesitate  to  suggest  that  a  large  number  of  smart  documents  can  be 
built  in  the  near  future  to  impact  a  wide  number  of  jobs.  Many  lETMs  for  mature  systems  will  lack 
the  smart  performance  support  features  and  will  not  be  electronically  integrated  into  a  total 
maintenance  system. 

4.2  Implications  for  the  Organizations 

In  the  broad  sense,  technology  is  not  only  hardware  and  software.  It  is  knowledge  and 
techniques  about  how  to  accomplish  a  goal.  These  have  implications  for  coordinating  task 
performance  within  organizations.  One  relevant  way  to  characterize  technological  impacts  on 
organizations  (Kast  &  Rosenzweig,  1974)  is  to  consider  three  aspects  that  affect  the  organizations 
coordination  structure.  These  characteristics  are  linking,  mediating,  and  intensive.  A  technology  is 
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linked  when  there  is  a  serial  interdependence  between  units  of  work  activities.  An  assembly  line  is 
the  classic  case  of  linked  technology.  A  mediating  technology  is  one  that  requires  an  interchange 
among  individuals  who  are  otherwise  independent.  Home  television  shopping  is  a  new  mediating 
technology  between  consumers  and  those  who  have  products  to  sell.  An  intensive  technology  is 
one  that  requires  use  of  a  variety  of  techniques,  perhaps  from  different  specialties,  in  order  to 
achieve  goals. 

lETM  technology  and  the  integration  of  lETMs  with  training  and  education  represent  all 
three  of  these  characteristics.  There  is  an  interdependence  in  the  flow  of  maintenance, 
documentation  and  training  activities  so  the  technology  is  linked.  It  is  mediated  because,  in  each 
stage  of  the  life  cycle  for  several  products,  developmental  personnel  must  interface  with  personnel 
in  other  stages  and  within  stages,  as  well  as  with  the  end-user.  It  is  intensive  because  it  draws  upon 
several  disciplines:  maintenance,  logistics,  documentation,  and  training  each  with  both  unique  and 
common  basis  for  technology.  The  integration  of  lETM  technology  into  training  and  education, 
therefore,  requires  a  look  at  the  coordination  roles  and  functions  of  organizations. 

There  appears  to  be  evidence  that  training  and  documentation  organizations  will  need  to 
change  at  several  levels.  Typically  in  the  past,  large  organizations  inside  and  outside  the  military 
have  had  one  unit  which  took  care  of  documentation  and  one  responsible  for  developing  and 
operating  training.  Because  of  the  impact  on  both  areas  it  will  be  necessary  for  even  greater 
coordination  between  the  agencies  and,  most  likely,  an  integration  of  the  agencies  at  least  for  the 
purpose  of  developing  and  managing  lETMs.  This  will  be  especially  true  if  the  path  of  an 
integrated  data  base  is  taken. 

We  have  already  discussed  the  expected  changes  in  how  training  is  conducted  in  the 
classroom  at  the  schoolhouse  level.  We  expect  that  this  will  also  impact  the  schoolhouse 
organization  as  instructors  become  managers  of  learning  resources  more  than  authorities  in  content 
that  have  been  trained  to  lecture  and  demonstrate.  There  is  also  an  impact  on  course  management 
and  the  organizations  responsible  for  specifying,  developing,  and  evaluating  those  courses.  It  is 
beyond  the  scope  of  this  effort,  and  the  abilities  of  the  authors,  to  redesign  the  organizations 
involved.  We  offer  some  further  thoughts  on  organizational  elements  and  coordination  below  based 
on  discussions  with  people  in  the  survey. 

4.2.1  Coordinated  Definition  and  Development  of  DETMs 

In  most  cases  it  seemed  that  the  training  organizations  who  would  be  receiving  BETMs  were 
involved  at  least  at  a  cursory  level  in  the  definition  and  development  of  lETMs.  In  some  cases, 
schoolhouse  personnel  played  a  part  in  the  development.  However,  more  often  it  appeared  that  the 
definition  of  the  class  of  lETM,  especially  in  new  weapon  systems,  and  the  resulting  impact  on 
training  requirements  was  not  a  systematically  coordinated  effort.  It  is  also  unclear  how  logistic 
task  analysis  and  Navy  Training  Plan  development  play  a  part  in  the  definition  and  design  of 
lETMs.  To  date,  it  appears  that  the  same  paper  documentation  policies  and  procedures  are  being 
followed. 
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4.2.2  Course  Management 


With  the  rapid  evolution  in  technologies  and  advances  in  system  capabilities,  the  analysis 
of  management  requirements  should  be  viewed  as  an  ongoing  process  both  during  and  after 
implementation.  In  some  cases,  the  changes  in  management  requirements  may  simplify 
procedures,  reduce  time  and  effort,  or  make  more  efficient  use  of  facility  space.  In  other  cases,  the 
changes  may  add  a  number  of  new  functions  to  be  performed  and  requirements  to  be  supported, 
require  personnel  training  for  new  skills,  dictate  a  different  mix  of  personnel,  or  increase  time  and 
effort.  All  of  these  changes  have  cost  implications  that  must  be  addressed  as  part  of  the  decision 
making  and  planning  process. 

Implementation  of  the  new  technology  in  just  one  course  in  one  classroom  in  a  schoolhouse 
can  have  consequences  that  extend  beyond  that  classroom.  For  the  most  part,  the  management 
requirements  for  the  lETM  based  electronic  classroom  are  very  different  from  a  traditional 
classroom  and  have  implications  for:  (1)  the  school/course  management  organization  and  its 
structure,  policies,  and  procedures,  (2)  the  instructors  and  their  duties,  (3)  the  roles  and 
responsibilities  of  management,  administrative,  and  support  personnel,  and  (4)  the  facilities  and 
their  utilization.  NAVEDTRA  135,  Navy  School  Management  Manual,  defines  the  processes, 
procedures  and  documentation  required  for  management  of  NAVEDTRACOM  schools.  It  is 
expected  that  as  greater  coordination  of  efforts  among  logistics,  maintenance,  documentation  and 
training  communities  is  achieved  the  policies  and  procedures  set  forth  in  that  document  will  be 
revised. 

4.2.3  Planning 

Although  adequate  planning  is  essential,  the  process  normally  required  for  planning  and 
implementing  such  technology  is  too  long  and  time  consuming.  Often  times  the  technology  is 
obsolete  before  the  system  is  installed.  Management  tools  need  to  be  developed  to  support  the 
planning  and  budgeting  phase  and  facilitate  the  acquisition  and  installation  phase.  For  example, 
NAVSEA  has  tasked  personnel  at  the  Fleet  Technical  Support  Center,  Atlantic  (FTSC-LANT)  to 
develop  a  generic  Equipment  Facilities  Requirement  Plan  (EFRP)  and  a  generic  Abbreviated 
System  Decision  Paper  (ASDP)  for  use  in  planning  and  initiating  future  implementations.  The 
purpose  of  these  generic  documents  is  to  serve  as  guidelines  identifying  the  basic  requirements  and 
indicating  where  school  specific  information  needs  to  be  furnished. 

Likewise,  personnel  from  the  Naval  Sea  Systems  Command  Automated  Data  Systems 
Activity  (SEAADSA),  the  Hardware  Installation  Activity  for  the  Gas  Turbine  BETM 
implementation  at  Service  School  Command,  Great  Lakes  (SSC  GLARES),  discussed  the 
requirement  to  streamline  the  process  for  procuring  the  electronic  classroom  components.  Efforts 
are  underway  to  establish  a  delivery  order  type  contract  to  allow  schools  to  acquire  COTS 
hardware  and  software  (i.e.,  the  LINK  System  and  the  LiveBoard)  that  are  specified  as  part  of  the 
configuration  and  are  not  available  under  the  current  small  multi-user  computer  contract  or  GSA 
schedules.  This  type  of  contract  may  include  provisions  for  quantity  buys,  spares,  installation  and 
maintenance  support,  and  user  training.  It  should  result  in  lower  acquisition  costs  and  reduce  the 
lead  time  to  order  and  install  the  equipment. 
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Adequate  planning  is  key  to  successful  implementation  of  the  lETM/AC  technology.  As 
documented  in  a  paper  presented  by  the  Gas  Turbine  (GS)  Technical  Training  Officer,  LT.  Eric 
Fretz  (1995),  the  speed  with  which  the  technology  was  implemented  in  the  GS  School  led  to  some 
early  mistakes  that  could  be  avoided  by  more  planning  and  discussion.  For  example,  the  size  of 
automated  classrooms  was  decreased  to  a  maximum  of  12  to  15  students  from  the  maximum  of  25 
students  in  a  paper-based  classroom.  The  classroom  size  has  implications  for  supporting  the 
projected  student  throughput  as  well  as  classroom  scheduling  and  system  resource  management. 
To  meet  projected  throughput  requirements,  the  electrical  GS  school  has  three  classrooms  with  12 
student  stations  each,  the  mechanical  GS  school  has  one  classroom  with  15  student  stations,  and 
both  schools  share  one  classroom  with  12  student  stations. 

Also,  the  initial  SSC  GLAKES  hardware  and  software  configuration  did  not  include  the  use 
of  a  local  area  network  (LAN)  to  link  the  instructor  station  (server)  and  the  student  stations  (clients) 
in  the  classroom.  When  this  requirement  was  recognized,  an  EtherNet  LAN  and  Windows  for  Work 
Groups  3.11  were  installed.  However,  this  simple  setup  does  not  provide  the  necessary  capabilities 
and  security  features  so  plans  are  currently  being  made  to  upgrade  to  Windows  NT  software. 
Physical  security  of  the  classrooms  and  management  of  the  COTS  software  inventory  are  major 
concerns  that  need  to  be  addressed  in  addition  to  maintaining  the  integrity  of  the  instructor  station 
configuration  and  the  security  of  the  course  tests. 

4.2.4  Life  Cycle  Support 

Life  cycle  management  and  support  is  impacted  by  the  technology  and  requires  new 
organizational  elements  from  both  training  and  acquisition  commands  For  example,  NAVSEA  has 
recently  tasked  the  several  organizations  to  define  the  life  cycle  management  and  support 
requirements  for  the  five  electronic  classrooms  at  the  GS  schools.  The  specification  of  life  cycle 
support  requirements  for  the  lETM  electronic  classrooms  is  evolving  as  problems  and  potential 
solutions  are  identified. 

SEAADSA  was  designated  as  the  Hardware  In-Service  Engineering  Activity  (ISEA)  and 
the  Basic  Software  ISEA  for  the  classrooms.  The  term  hardware  is  broken  down  into  two 
definitions.  Computer  hardware  is  defined  as  any  piece  of  physical  equipment  including 
computers,  monitors,  desks,  and  chairs.  Non-computer  hardware  is  defined  as  any  piece  of  physical 
equipment  related  to  hotel  services  (i.e.,  lighting,  electrical  power,  heating/air  conditioning)  that  is 
necessary  for  the  proper  operation  of  the  lETM  electronic  classroom  configuration.  Basic  software 
is  defined  as  any  computer  program  or  collective  group  of  programs  that  contribute  to  the  operation 
of  the  lETM/AC  (i.e.,  Windows  and  other  standard  software  that  may  be  loaded  on  the  system  at 
the  time  of  purchase).  SEAADSA  is  responsible  for  coordinating  any  changes  in  the  hardware  and 
basic  software  configuration  for  the  GS  classrooms. 

The  Naval  Sea  Logistics  Center,  Code  30  (SEALOG)  was  designated  as  the  Hardware  and 
Basic  Software  Life  Cycle  Manager  (Hardware  LCM).  Personnel  are  in  the  process  of  preparing  a 
Life  Cycle  Management  Plan  that  documents  the  responsibilities  and  tasks  for  supporting  the 
classrooms  over  their  life  cycle.  FTSC-LANT  was  designated  as  the  Software  ISEA.  Personnel  are 
in  the  process  of  preparing  a  Software  Life  Cycle  Management  Plan  that  documents  the 
responsibilities  and  tasks  for  supporting  the  software  configuration.  SSC  GLAKES,  as  the  end  user 
of  the  lETM  electronic  classroom  technology,  is  responsible  for  monitoring  the  condition  of  the 
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hardware  and  software  configuration,  ensuring  appropriate  maintenance  actions  are  taken,  and 
coordinating  with  SEAADSA,  SEALOG,  and  FTSC-LANT  on  life  cycle  support  issues  for  the 
classrooms. 

Based  on  the  GS  school  experience,  it  is  clear  that  implementation  of  the  technology 
imposes  additional  personnel  support  requirements  on  the  schoolhouse.  Some  personnel  support  is 
provided  by  vendors  and  contractors  involved  in  the  effort.  However,  most  of  the  personnel  support 
for  the  automated  classrooms  comes  from  the  school  managers,  course  supervisors  and  instructors 
with  assistance  from  the  New  Technologies  Division  staff.  Therefore,  the  GS  “C”  School  managers 
are  preparing  an  ASDP  for  submission  to  the  Naval  Education  and  Training  Program  Management 
Support  Activity  (NETPMS A)  to  address  the  additional  support  requirements  over  the  life  cycle  of 
the  lETM  electronic  classrooms. 

4.2.5  Instructor,  Student,  and  Staff  Ikaining 

Only  two  military  lETM  apphcations,  shown  earlier  in  Table  2,  had  enough  experience 
with  integration  of  lETMs  into  a  formal  training  program  to  identify  what  was  needed  for  both 
preparing  instructors  and  students  to  successfully  use  the  technology  in  the  classroom.  The  two 
schools  were  the  Army’s  Air  Defense  School,  6th  Brigade  at  Ft.  Bliss,  which  covers  Patriot  Missile 
military  occupational  specialties  (MOS),  and  the  Gas  Turbine  Technician  courses  at  the  Service 
School  Command,  Great  Lakes.  In  both  cases,  the  lessons  learned  were  clearly  similar  and 
significant.  Instructors  need  a  formal  training  course.  Students  need  training  to  use  the  lETM  and 
any  associated  electronic  classroom  technology  embedded  into  their  regular  course  of  instruction. 
Basic  computer  literacy  in  a  Microsoft  Windows  environment  cannot  be  assumed  to  be  the  norm. 

The  training  at  Ft.  Bliss  is  conducted  by  the  U.S  Army  Missile  Command,  Integrated 
Material  Management  Center  (IMMC).  IMMC  also  supports  the  IADS  (Interactive  Authoring  and 
Display  System)  used  to  develop  and  present  missile  command  lETMs.  The  same  authoring 
software  is  used  in  the  schools  by  instructors  to  develop  and  personalize  plans  of  instruction 
(POIs).  Therefore,  although  the  IMMC  courses  are  described  as  authoring  courses,  they  cover 
some  of  the  same  content  that  will  be  described  below  for  the  instructors  at  Great  Lakes.  IMMC 
offers  courses  ranging  from  a  two  day  authoring  overview  to  a  week-long  detailed  authoring 
tutorial.  Since  IMMCs  IADS  customers  may  be  located  anywhere  the  courses  are  offered  at 
IMMCs  home  base  at  Redstone  Arsenal  or  at  the  customers  facilities. 

The  recent  course  offered  for  Patriot  Missile  instructors  provided  105  Senior  NCO,  NCO 
and  civilian  instructors  with  a  two  and  one-half  day  basic  course  on  the  use  of  IADS  to  author  and 
link  POIs  into  both  lETMs  and  instructor  produced  materials  from  the  POIs.  Training  was  provided 
on  how  to  navigate  the  lETMs  and  find  the  names  of  text  units  and  graphics.  Material  developed 
by  COTS  tools,  such  as  the  schools  use  of  Asymmetric  Toolbook,  can  be  launched  by  IADS  as 
indicated  in  the  POIs.  Training  for  use  of  COTS  development  tools  was  not  included  in  this  course. 
The  instruction  included  linking  to  graphics,  multilinking  from  text  to  text  passages,  and 
conditional  branching  to  modify  displays.  Instructor  specific  techniques,  as  opposed  to 
standardized  POI  links  required  by  everyone,  for  incorporating  materials  into  the  courses  was  also 
covered.  All  of  these  instructor  activities  in  the  POI  lesson  plans  are  invisible  to  the  students,  who 
also  have  the  lesson  plans  at  their  stations.  The  display  to  students  occurs  at  their  stations  as  well 
as  on  a  large  screen  display  under  instructor  control  via  his/her  POI. 
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The  course,  at  two  and  one-half  days  was  found  to  be  of  adequate  length  for  those 
instructors  who  were  proficient  in  the  use  of  Microsoft  Windows  However,  others  had  trouble.  In 
fact,  IMMC  has  found  that  here  is  a  large  cultural  hurdle  for  the  older  Senior  NCOs  who  have  spent 
many  years  as  podium-based  instructors.  They  may  see  lETM  technology  as  a  threat  and  may  also 
be  somewhat  computer-phobic.  Younger  instructors  seemed  to  have  less  trouble  generally,  but  this 
also  posed  a  threat  to  Senior  NCOs.  It  appears  that  the  course  assumes  a  prerequisite  level  of 
proficiency  in  Microsoft  Windows.  Those  not  familiar  with  Windows-based  systems  need 
instruction  on  all  of  the  basics  of  operating  the  computers.  The  course  instructors  have  attempted 
to  include  such  instruction,  but  there  is  insufficient  time  to  bring  those  personnel  up  to  the  level 
needed. 


Student  training  on  the  use  of  the  lETM  classroom  technology  was  also  required  in  the 
Patriot  Missile  courses.  It  has  been  incorporated  in  an  ongoing  manner,  as  opposed  to  up  front,  in 
the  courses.  Most  students  lack  basic  computer  literacy  skills.  This  also  has  meant  that  efforts  need 
to  be  made  to  teach  instructors  to  train  the  students  in  these  basic  computer  skills.  As  expressed 
during  one  interview  with  an  IMMC  representative  it  would  be  highly  desirable  for  students  to 
have  been  given  these  skills  early  in  their  Army  training,  perhaps  before  initial  MOS  training. 
Instructors  at  the  Patriot  Missile  school  have  found  that  management  of  the  students  is  impossible 
without  having  a  large  screen  display  to  help  guide  the  students  through  the  exercises  they  are  to 
complete  during  class.  In  addition,  it  is  typical  to  have  two  instructors  so  that  one  can  walk  around 
the  classroom  looking  for  problems  and  providing  individual  guidance  as  may  be  needed.  Thus, 
tools  for  instructor  management  of  students  can  go  beyond  support  for  learning  the  content  area 
and  include  support  for  learning  basic  computer  skills. 

In  the  Gas  Turbine  courses  at  SSC  GLARES,  the  level  of  computer  literacy  among  students 
is  also  variable  and  a  large  percentage  of  students  do  not  have  sufficient  basic  Windows  computer 
skills.  On  the  first  day  of  class,  students  receive  approximately  six  hours  of  training  on  Windows- 
based  skills  required  for  the  lETM  and  electronic  classroom.  The  training  varies  depending  on  the 
students  familiarity  with  computers.  Novices  are  directed  to  go  through  the  mouse  and  Windows 
tutorials  before  being  introduced  to  the  lETM  on-line  help  index  and  practicing  navigating  through 
the  lETM.  Students  concentrate  on  searching  the  database,  making  annotations,  and  creating 
bookmarks.  Instructors  report  that  most  students  are  comfortable  with  the  system  and  proficient  in 
using  it  by  the  end  of  the  first  week  of  class.  Students  that  continue  to  have  trouble  may  receive 
additional  assistance  during  the  voluntary  study  time. 

Students  have  the  capability  to  use  the  annotation  feature  in  the  lETM  database  to  take 
notes  on-line  at  their  work  stations  during  class  sessions.  However,  there  appear  to  be  several 
drawbacks  to  on-line  note  taking.  Many  students  do  not  have  basic  typing  skills  and  hunting  and 
pecking  on  the  keyboard  disrupts  their  concentration  on  the  course  content.  Since  the  only  systems 
available  for  students  to  review  their  notes  on  floppy  disk  are  the  ones  in  the  lETM  classrooms, 
their  study  time  is  restricted  to  the  afternoon  hours.  They  do  not  have  access  to  systems  after  hours 
or  over  weekends.  Likewise,  the  students  may  not  be  able  to  review  their  notes  on-line  after  they 
return  to  their  jobs  in  the  fleet. 

To  maximize  the  effectiveness  of  the  lETM  and  electronic  classroom  technology  at  SSC 
GLARES,  the  instructors  must  be  adequately  trained  on  the  operational  features  and  enhanced 
capabilities  of  the  technology.  Since  the  instructional  staff  has  varying  degrees  of  computer  literacy 
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and  experience,  the  initial  training  was  provided  on  an  ad  hoc  basis  by  the  instructors  with 
computer  experience  who  were  willing  to  experiment  with  the  system.  Based  on  this  experience, 
the  course  managers  at  SSC  GLARES  have  developed  a  40  hour  syllabus  for  initial  instructor 
training  on  how  to  use  the  technology  in  the  classroom.  This  syllabus  will  be  used  to  train  incoming 
instructors  in  the  mechanical  technician  courses  and  it  is  being  modified  to  train  electricd 
technician  course  instructors.  Instructors  are  sent  through  the  Windows  tutorials  if  they  do  not  have 
the  basic  computer  skills. 

The  instructor  lETM  course  covers  use,  operation  and  basic  concepts  of  the: 

•  Instructor  station. 

•  Student  station. 

•  Classroom  network  capabilities. 

•  LINK  system  incorporated  into  the  classrooms. 

•  Xerox  Liveboard  large  display. 

•  \fideo  (VCR)  support. 

•  Data  base  management  for  the  testing  system  incorporated. 

lETM  navigation,  including  launching  of  animations,  browsing  and  searching,  using 
bookmarks,  annotations,  and  use  of  the  view  packages  for  guiding  through  the  lETM  in  a  preset 
mode  is  also  taught.  Note  that  little  is  taught  on  authoring,  per  se,  in  the  course.  There  is  no 
equivalent  of  the  Army  IADS.  COTS  is  used  for  developing  graphics  and  animations  and  the 
lETMs  are  presented  via  COTS  viewers.  Members  of  the  instructional  staff  require  training  in 
COTS  applications  software  that  is  used  to  author  the  instructional  materials  and  develop 
multimedia  training  aids.  Instructor  authoring  training  should  be  considered  to  reduce  the  learning 
curve  and  increase  productivity. 

Currently,  the  authoring  skills  are  learned  by  instructors  on  their  own,  through  more 
experienced  instructors  and  developers  in  the  school.  Most  techniques  and  tools  used  seem  to  be  a 
function  of  the  energies  of  the  instructors.  They  need  to  be  involved  in  the  process  of  creating  the 
end  product  for  use  in  the  classroom.  Instructors  are  the  primary  source  of  innovation  and  key  to 
the  success  of  the  use  of  the  system;  including  modifications  of  the  Instructor  Guides  and 
incorporation  of  new  electronic  materials.  As  part  of  the  lessons  learned,  weekly  instructor  training 
programs  are  scheduled  to  share  corporate  knowledge,  float  ideas,  and  demonstrate  proven 
concepts  (Fretz,  1995). 

Other  staff  training  is  critical  for  successful  implementation  of  the  classroom  with  lETM- 
based  technology.  Appropriate  training  must  be  provided  for  course  management  and  support 
personnel.  The  need  for  qualified  personnel  to  perform  system  administration  tasks  and,  especially, 
network  administration  tasks  cannot  be  emphasized  enough.  As  stated  by  Fretz  (1995): 
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For  now,  the  resident  “computer  geek”  simply  learns  on  the  job  and 
from  vendors/installers.  TTiis  corporate  knowledge  is  perishable  and 
puts  the  school  up  against  a  rock  when  the  military  member  transfers.  It 
is  possible  to  ameliorate  this  situation  with  a  GS/contracted  network  ad¬ 
ministrator,  but  continual  training  programs  are  a  must.  (pg.  2) 

If  qualified  civilian  or  contractor  personnel  are  not  hired,  then  formal  training  programs  in 
microcomputer  hardware  and  software  configuration  management  and  certification  as  a  network 
administrator  are  essential  for  military  personnel  assigned  those  duties. 

4.3  Implications  for  Embedding  Training  Content  in  lETMs 

A  common  description  of  lETMs,  across  all  branches  of  the  services,  included  statements 
that  referred  to  the  embedding  of  training  into  lETMs.  That  is,  the  developers  and  sponsors  of 
lETMs  typically  think  in  terms  of  including  training  within  the  lETM  document  for  access  by  the 
user  on-demand.  While  this  is  a  somewhat  appealing  benefit  at  first  hearing  we  think  there  is  an 
important  issue  to  be  discussed.  The  question  is,  what  training  should  be  included  with  lETMs  and 
when  should  it  be  available.  A  primary  question  is  should  the  content  of  lETMs  include  formal 
instructional  materials  embedded  into  the  documents?  If  so,  should  instructional  managers  or 
instructors  be  involved  in  their  design  along  with  technician  experts?  What  is  the  source  for  the 
content?  What  is  its  relationship  to  the  curriculum  management  and  surveillance  requirements  of 
NAVEDTRA  135?  Initial  efforts  to  combine  training  materials  with  the  technical  document  itself, 
as  opposed  to  an  integrated  data  base  from  which  aU  materials  can  be  generated,  do  not  seem  to 
have  had  any  advantages  and  may  have  been  a  detriment  to  being  able  to  update  the  training 
materials  independent  of  the  lETM  document. 

One  example  of  embedded  training  is  the  GS  lETM.  In  this  case  the  formal  school 
Instructor  Guides  (IGs)  and  Trainee  Guides  (TGs)  were  revised  with  SGML  tagging  to  launch  and 
access  parts  of  the  lETMs  and  placed  on  the  lETM  CD-ROM.  Several  difficulties  occurred.  First, 
the  guides  are  now  difficult  to  revise  separately  from  the  lETM  revision  cycle.  The  GS  course  IGs 
and  TGs  authored  using  AIM  were  integrated  into  the  lETM  database  and  the  link  with  AIM  was 
lost.  The  links  between  lETM  databases  and  other  major  Navy  systems  like  AIM  need  to  be 
analyzed  to  allow  data  from  diverse  sources  to  be  dynamically  managed.  The  schoolhouses  need 
to  retain  control  of  the  IGs  and  TGs  and  the  course  revision  process  needs  to  be  independent  of  the 
lETM  development  and  revision  process. 

A  second  problem  was  that  the  student  has  access  to  the  IG  if  the  CD-ROM  was  used  for 
distribution.  The  school  therefore  reconfigured  the  student  stations  to  download  and  store  only  the 
pertinent  information  rather  than  use  the  CD-ROM.  Third,  the  requirements  for  course 
configuration  management  set  forth  by  NAVEDTRA  135  were  bypassed  and  the  course  is 
unapproved.  Given  that  hindsight  is  a  powerful  tool  for  wisdom,  we  do  not  wish  to  criticize  the 
developers  of  those  lETMs  and  the  school  for  what  seemed  reasonable  at  the  time.  However,  it  is 
now  a  lesson  learned  of  what  additional  considerations  are  needed  in  thinking  about  integration  of 
training  and  technical  information. 

A  more  effective  solution  seems  to  be  the  integration  of  data  bases  through  common 
definition  of  technical  and  training  data  elements.  This  common  basis  for  application  development 
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can  then  be  structured  and  managed  to  the  benefit  of  all  in  parallel,  but  allow  asynchronous 
application  development  and  updating. 

4.4  Integration  of  IVaining  Data  with  Technical  Information 

In  discussions  with  members  of  the  logistics,  maintenance,  publications  and  training 
communities,  we  often  encountered  a  thread  of  conversation  that  defined  both  technical  and 
training  information  as  data  elements.  Specifically,  many  voiced  the  opinion  that  these  data 
elements  can  be  better  defined  and  integrated  to  serve  the  goal  of  more  effective  maintenance, 
operations  and  training.  There  were,  however,  differences  in  how  to  accomplish  this.  While 
everyone  recognized  that  the  technical  information  could  be  drawn  for  use  in  training,  the 
questions  of  how  to  use  that  information  and  how  to  integrate  training  with  lETMs  varied  with  the 
perspective  of  the  organization  the  individual  represented.  Several  options  are  discussed  below. 

4.1.1  Use  of  lETMs  as  Ttaining  Media 

From  a  training  developers  viewpoint,  lETMs  can  be  considered  as  one  of  the  media 
resources  available.  Thus,  they  are  treated  as  any  other  resource,  such  as  text,  video,  graphics, 
devices  for  laboratory  activity.  Through  media  analysis  and  instructional  design  activities,  the 
developer  identifies  how  and  when  the  lETMs  are  to  be  incorporated  into  the  sequence  of  other 
instruction  needed  to  meet  learning  objectives.  However,  the  training  materials,  including  the  IG 
and  TG,  would  be  kept  separate  from  the  lETMs.  In  this  case,  the  lETMs  would  be  launched  by 
the  instructor,  the  IG,  or  even  interactive  courseware  (ICW)  designed  to  interface  with  the  lETMs 
to  allow  activities  such  as  troubleshooting  in  laboratory  exercises. 

4.4.2  Use  of  lETM  Data  as  a  Source  for  Developing  Training 

Another  way  to  think  of  lETMs  is  as  a  source  of  data  for  developing  training  materials. 
Several  of  the  organizations  surveyed  pointed  out  that  the  lETM  data  base  of  technical  information 
contains  the  same  information  needed  to  produce  CBT.  In  fact,  it  is  typical  when  developing  any 
instructional  material  of  a  technical  nature  to  draw  from  the  job  documents,  when  they  are 
available,  for  source  data.  Similarly,  it  is  hoped  by  lETM  end  user  organizations  in  the  training 
world  that  CBT  can  be  developed  with  automated  aids  from  an  lETM  database. 

While  most  efforts  are  either  only  conceptual  or  in  the  planning  stages,  one  Air  Force 
research  agency,  Armstrong  Laboratory,  and  its  contractor,  Mei  Technology  Corporation,  have 
achieved  some  success  along  these  lines  (Spector,  Arnold,  &  Wilson,  1995).  The  effort  makes  use 
of  a  prototype  system,  XAIDA  (Advanced  Instructional  Design  Advisor),  that  was  developed  to 
automatically  generate  CBT  courseware  from  information  input  by  a  subject  matter  expert  (Walsh 
&  Daly,  1995;  Wilson,  1995).  In  this  case,  the  researchers  identified  lETM  database  elements 
applicable  to  training,  using  an  lETM  for  the  F-16,  and  then  developed  software  to  parse  the 
information  for  input  to  XAIDA.  To  date  nomenclature,  location,  parts,  and  task  steps  for  testing 
and  repairs  have  been  parsed  with  some  success  and  input  into  XAIDA  for  lesson  generation. 
According  to  research  reports,  the  lessons  generated  are  neither  complete  nor  polished,  but  they 
can  then  be  used  by  a  subject  matter  expert  to  quickly  complete  the  lesson.  The  savings  in  time  to 
develop  CBT  was  reported  to  be  on  the  order  of  hundreds  of  labor  hours  per  hour  of  instruction 
since  a  sample  lesson  was  generated  in  about  ten  minutes. 


52 


4.4.3  Common  Data  Elements  and  Integrated  Databases 

Given  the  cost  of  data  development  both  for  manuals  and  training  and  the  commonality  of 
information  used  in  both  applications,  it  seems  reasonable  to  consider  the  possibility  of  defining 
the  data  elements  so  that  such  diverse  applications,  as  well  as  other  logistics  applications,  could 
reuse  the  same  data  with  ease  and  efficiency.  The  goals  would  be  to  develop  data  once  and  to  define 
it  in  such  a  way  as  to  facilitate  electronic  data  interchange  among  applications.  There  are  several 
movements  in  progress  that  make  this  more  feasible  now  than  in  previous  years.  Although,  all  of 
the  pieces  are  not  in  place,  it  is  worth  discussing  the  efforts  that  are  underway  that  may  influence 
the  possibility  of  achieving  a  common  database. 

First,  we  note  that  at  a  top  level  of  DoD  the  CALS  program  has  as  one  of  its  initiatives  the 
development  of  an  integrated  weapon  system  database.  The  technical  information  in  this  database 
would  include  common  data  elements  in  use  across  weapon  systems  and  most  likely  would  have  a 
great  deal  of  the  information  needed  for  lETMs  including  equipment  characteristics,  logistic 
analysis  of  tasks  to  be  performed,  procedural  element  of  the  tasks,  and  definitions  of  operator  and 
maintenance  personnel.  In  recognition  of  such  goals  as  those  of  CALS  and  others,  the  Defense 
Information  Systems  Agency  (DISA)  has  been  set  up  as  the  DoD  steward  for  DoD  data 
standardization.  The  goal  by  DoD  of  data  standardization  and  administration,  as  stated  in  DoD 
Directive  8320.1,  is  to: 

a.  Support  DoD  operations  and  decision  making  with  data  that  meets  the 
need  in  terms  of  availability,  accuracy,  timeliness,  and  quality. 

b.  Structure  the  information  systems  in  ways  that  encourage  horizontal, 
as  well  as  vertical,  sharing  of  data  in  the  Department  of  Defense,  and 
with  other  government  agencies,  private  sector  organizations,  and  allied 
nations,  consistent  with  national  security  and  privacy  requirements. 

(p.3) 

To  achieve  these  goals,  specific  procedures  for  data  element  standardization,  DoD  8320.1- 
M-1,  and  data  administration,  DoD  8320. 1-M,  have  been  defined  by  DISA. 

Another  recent  occurrence  is  even  more  directly  related  to  providing  a  basis  for  definition 
of  training  data  elements  for  an  integrated  database  and  enhancing  capability  for  electronic  data 
interchange.  In  June,  1995  the  decision  was  made  by  the  Defense  Standardization  Improvement 
Council  (DSIC)  to  convert,  by  June  of  1996,  MIL- STD  1379D  to  an  acquisition  guide  with 
specifications  for  training  data  required.  DSIC  is  the  office  established  by  Defense  Secretary  Perry 
to  oversee  the  process  of  standards  and  specifications  conversion.  The  conversion  is  part  of  the 
Secretary’s  direction  toward  overhauling  these  documents  with  the  view  of  doing  business  in 
accordance  with  best  commercial  practices  and  minimizing  government  oversight  functions. 

In  response  to  the  Secretary’s  direction,  a  Defense  Training  Standards  Working  Group 
(DTSWG)  has  been  established  with  Naval  Air  Systems  Command  PM  A  205-4B  as  both  the 
preparing  activity  and  as  the  chairperson  (Cogan,  1995).  The  DTSWG,  supported  by  contract 
efforts,  is  now  working  toward  definition  of  training  data  elements.  An  IDEF  (Integrated 
Computer-Aided  Manufacturing  Definition)  training  data  model,  in  conformance  with  the  DISA 
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procedures  for  data  element  standardization,  is  scheduled  for  completion  by  December  of  1995. 
The  converted  MIL- STD  1379D  is  expected  to  have  training  data  entity  and  attribute 
specifications,  an  IDEF  IX  logical  model  view,  a  user  view,  and  product/data  eleirient  applicability 
cross  reference  (p.  2).  Upon  completion  of  this  effort  it  might  be  expected  that  these  definitions  will 
be  the  basis  for  submittal  to  DIS  A. 

However,  what  is  missing  is  a  similar  effort  by  the  lETM  Tri  Service  Working  Group  or  an 
organization  under  their  direction.  A  valuable  step,  from  the  standpoint  of  the  training  community, 
toward  database  integration  and  electronic  data  interchange  would  be  taken  by  that  work. 
Certainly,  there  are  those  who  would  like  to  draw  from  the  lETM  databases  to  generate  CBT 
(Doman,  1995;  Robinson,  1995)  and  other  instructional  material,  as  demonstrated  by  the  Air  Force 
XAIDA  project  described  above.  It  is  likely  that  applications  could  make  use  of  the  power  of  an 
integrated  relational  database  as  well  as  the  SGML  tagged  lETM  database.  It  should  be  noted  that 
although  the  DTDs  define  data  elements  and  entities  for  a  given  lETM  application  it  was  stated  by 
several  of  those  interviewed  in  this  survey  that  although  two  different  lETM  applications  comply 
with  the  DETM  specifications  they  can  not  talk  to  each  other.  The  SGML  databases  are  not 
compatible  with  each  other  and  can  not  be  queried  by  another  application  for  extraction  of  data 
with  the  same  format,  technique,  and  software. 

One  effort  that  may  be  immediately  addressing  the  access  of  lETM,  as  well  as  other  logistic 
data,  is  the  AIM  I  and  n  software  developments  going  on  under  the  auspices  of  the  Software 
Support  Office  (SSO)  at  the  Naval  Air  Warfare  Center/Training  Systems  Division  (NAWC/TSD). 
Several  of  our  conversations  with  personnel  outside  of  that  organization  indicated  that  it  is  intended 
that  AIM  access  lETM  data  in  SGML  form  for  development  of  instructional  material,  derive  lETM 
element  and  entity  identifiers  for  inclusion  in  the  Related  Instructor  Activity  column  of  the 
Instructor  Guide  and  map  the  AIM  DTDs  to  the  lETM  DTDs.  We  have  been  unable  to  confirm  this 
directly  with  NAWC/TSD  to  date. 

4.5  Impact  on  Education 

In  our  survey,  we  found  that  educational  organizations,  including  institutions  of  higher 
education,  were  very  active  in  producing  and  using  hypertext/hypermedia  documents.  However,  in 
most  cases  these  were  focused  on  simply  the  linking  of  information  and  access  through  either  free 
or  guided  navigation.  That  is,  there  were  seldom  smart  features  used  to  solve  problems.  There  are 
several  interesting  exceptions  to  this.  For  example,  there  are  several  SGML-based  applications  that 
have  been  developed  to  support  scholarly  research  in  the  arts  and  humanities.  The  SGML 
structures  are  used  to  define  and  relate  content  in  ways  that  scholars  ask  questions.  However,  the 
question  needs  to  be  asked  whether  the  lETM  technology  has  broad  application  in  the  educational 
sectors.  We  believe  that  the  answer  is  yes,  especially  when  intelligence  is  put  into  the  document. 

As  defined  by  Siegel  (1993)  of  Indiana  University,  the  virtual  textbook  is  an  example  of 
how  intelligence  can  be  applied.  The  virtual  textbook  is  conceptualized  as  a  hypermedia 
application  (not  SGML  based)  that  combines  local  and  distributed  course  content  in  a  problem- 
centered  delivery  environment.  The  model  of  instruction  for  the  virtual  textbook  emphasizes 
teaching  about  problems  to  be  solved,  skills  to  solve  problems  and  providing  tools  for  accessing 
information.  This  is  opposed  to  a  focus  on  teaching  the  information  itself. 
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The  following  paraphrases  Siegel’s  description  of  old  and  new  models  of  instruction.  First, 
the  teacher  will  be  a  facilitator,  a  resource  provider,  and  will  not  be  expected  to  know  all  of  the 
information  and  deliver  it.  This  contrasts  with  the  current  model  of  teachers  as  disseminators  of 
information  with  full  control  of  the  classroom  learning  process.  Second,  the  student  will  be  an 
active  and  interactive  researcher,  problem  solver,  and  learning  strategist.  The  student  will  learn 
through  effective  use  of  information  and  problem  solving  techniques.  This  contrasts  with  the 
current  models  that  treat  the  student  as  a  receiver  of  information  and  test  taker  who  works  alone  to 
memorize  and  learn.  Third,  the  content  will  be  treated  as  interdisciplinary  rather  than  sets  of 
information  that  are  separated  and  organized  by  “disciplines”  or  artificial  units  The  point  here  is 
that  there  is  too  much  information  to  memorize.  The  goal  is  to  teach  students  how  to  access 
information,  determine  relationships,  use  it  to  solve  problems,  and  develop  multiple  perspectives 
on  the  information. 

lETM  technologies  can  be  used  to  support  such  goals  in  education.  As  discussed  earlier  in 
our  review  of  SGML  applications  in  higher  education,  we  see  similar  uses  of  SGML  and  lETM 
technologies  for  case-based  education  and  training  in  the  military.  There  are  many  military 
instructional  situations  where  it  is  more  appropriate  for  the  events  of  learning  to  be  adult,  learner- 
centered  rather  than  instructor-centered. 
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Section  Five 

Differences  Between  Job  Performance  Support 

Documentation  and  Training 


5.0  Differences  Between  Job  Performance  Support 
Documentation  and  Training 


A  major  issue  that  has  arisen  in  the  past  two  years,  and  is  gaining  strength  in  its 
significance,  is  how  much  should  lETMs  encompass  training?  That  is,  should  lETMs  include  part 
or  all  of  the  training  that  will  be  needed  throughout  the  career  of  a  end  user.  There  are  many 
implications  that  have  not  been  addressed  yet  regarding  this  central  issue.  For  example,  if 
instruction  is  included  in  lETMs  who  will: 

•  Define  it  as  a  requirement  and  continue  to  assess  that  requirement. 

•  Design  it  for  effectiveness,  taking  into  account  differences  in  target  population  charac¬ 
teristics. 

•  Design  it,  taking  into  account  media  selection  principles. 

•  Design  it,  taking  into  account  instructional  strategies  for  different  types  of  learning 
and  learners. 

•  Manage  the  configuration  from  a  basis  of  both  instructional  and  job  performance 
effectiveness. 

•  Manage  the  continuing  support  required  for  the  combinations  of  classroom,  fleet  and 
just-in-time  training. 

There  are  several  proponents  for  including  training  within  lETMs  on  the  grounds  that  the 
technical  information  database  can  be  the  same  and  that  the  same  technology,  such  as  SGML,  can 
benefit  both.  For  example,  as  stated  by  Alligood  (1994)  An  lETM,  or  more  appropriately  an 
Interactive  Electronic  Training,  Operations,  and  Maintenance  Support  System  (lETOMSS)is  a 
database  of  information  that  includes  and/or  provides  linkage  to  all  technical  information  to 
support  training,  operations  and  maintenance  for  a  set  of  systems  or  equipment.(p.  3) 

In  this  section,  we  will  describe  the  pros  and  cons  of  including  training  within  lETMs.  We 
will  also  argue  that  while  it  will  be  effective  and  efficient  to  have  one  technical  information 
database  for  both  applications  and  that  SGML  and  other  technologies  underlying  the  lETMs  can 
benefit  the  training  world,  there  are  two  different  goals  to  be  reached  and  it  is  not  proper  to  define 
the  same  structure  or  content  for  an  lETM  as  one  would  for  a  training  document.  We  will  further 
discuss  the  role  of  structures  and  navigation  for  job  performance  support  documents  like  lETMs 
versus  similar  documents  meant  to  instruct  and  allow  acquisition  of  learning.  Specifically,  these 
issues  will  be  illustrated  by  discussions  of  some  instructional  design  issues  for  hypertext  and  two 
of  the  media,  animation  and  video,  that  are  commonly  called  upon  for  inclusion  in  lETMs. 

5.1  Hypertext  and  Learning 

Our  thesis  concerning  hypertext  for  learning  is  that  it  should  be  structured  for  that  purpose 
and  that  to  do  so  in  an  lETM  could  thwart  the  goal  of  developing  the  lETM  as  an  effective  job 
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performance  support  aid  within  the  context  of  an  integrated  maintenance  system.  However,  before 
developing  that  argument  we  should  point  out  that  not  all  documentation  should  necessarily  even 
be  in  hypertext  format.  In  fact,  it  may  be  better  to  have  some  things  in  paper  format  rather  than 
electronic.  For  example,  if  we  consider  existing  classroom  instruction  it  may  not  be  worthwhile  to 
convert  Student  Guides  (SGs)  at  all.  In  other  cases,  parts  of  the  SGs  may  benefit  from  the  hypertext 
treatment.  The  instructional  analysis  and  design  process  should  determine  if  there  are  links  to 
information  worth  including  or  launches  to  lETMs  or  other  external  electronic  media  that  need  to 
be  incorporated.  In  such  cases  a  complete  hypertext/hypermedia  format  may  be  appropriate. 

The  criteria  should  be  based  upon  relative  effectiveness  between  media  forms.  It  is  a  fact 
that  the  paper,  linear  form  and  the  electronic,  hypertext/hypermedia  form  are  two  different  media. 
This  is  not  simply  a  matter  of  replacing  paper  in  the  classroom  or  providing  training  that  can  be 
more  in  line  with  the  goals  of  lETM  implementations.  The  paper  versions  of  SGs,  or  portions  of 
them,  may  have  some  advantages  over  hypertext  versions  that  will  have  to  be  evaluated  on  a  case- 
by-case  basis.  Shneiderman,  Kreitzberg,  and  Berk  (1991),  who  have  been  concerned  with 
computer-human  interfaces  for  the  last  several  years,  have  pointed  out  that  the  printed,  paper 
version  of  a  text  may  be  more  appropriate  at  times  than  hypertext  (Shneiderman,  Kreitzberg,  & 
Berk,  1991).  To  paraphrase  their  logic  the  printed  page; 

•  May  have  better  resolution  which  could  be  needed  for  color  graphic  s  or  some  other 
visual. 

•  Is  portable  and  can  be  taken  to  places  that  the  hypertext/hypermed  ia  might  not  (such 
as  out  of  the  classroom,  off  the  base,  and  off  the  ship). 

•  Is  convenient  to  make  notes  on  and  highlight  portions  for  study. 

Shneiderman,  et  al  suggest  that  there  are  “Golden  Rules  of  Hypertext”  to  be  applied  as  a 
test  for  hypertext  treatment.  These  rules  assume  there  is  a  large  body  of  information  organized  into 
numerous  fragments,  the  fragments  relate  to  each  other,  and  the  reader  needs  only  a  small  fraction 
at  any  time. 

However,  from  the  standpoint  of  training  effectiveness,  the  more  important  issue  is  the  need 
to  structure  text  for  a  learning  experience  as  opposed  to  a  job  aid.  To  do  so  requires  an  analysis  of 
the  student  and  the  content.  Although  it  is  certainly  possible  to  treat  a  paper  text  as  nonlinear  and 
jump  around  from  page  to  page,  it  is  probably  a  better  form  than  hypertext  to  ensure  that  dependent 
learning  sequences  are  followed,  especially  by  learners  who  may  not  be  good  at  learner  control. 

Specifically,  the  hypertext/hypermedia  treatment  allows  much  more  learner  control  with 
the  possibility  of  getting  “lost  in  hyperspace.”  There  is  research  to  indicate  that  hypermedia 
readers,  especially  novice  readers,  can  have  problems  in  accessing  and  sequencing  information  in 
a  large  database  (Gay  &  Mazur,  1991).  There  are  a  variety  of  techniques  to  assist  the  user  that  can 
minimize  reading  problems.  However,  since  it  can  happen,  it  raises  a  point  about  which  Navy 
students  may  be  suited  to  hypertext/hypermedia  materials  and  which  may  not.  One  recent  attempt 
to  identify  and  describe  the  effective  dimensions  of  interactive  learning  systems,  including 
interactive  multimedia  of  the  sort  that  could  be  developed  for  Navy  classrooms,  concluded  that 
learner  control  is  a  very  complex  issue  for  which  more  and  better  research  is  needed  (Reeves,  1992; 
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Reeves  &  Harmen,  1993).  That  certainly  applies  to  the  varieties  of  students  and  learning  tasks  in 
Navy  classrooms. 

One  recent  study  of  learning  with  hypertext  found  that  there  is  a  difference  in  learning 
between  students  with  low  prior  knowledge  and  high  prior  knowledge.  Hillinger  and  Leu  (1994), 
using  content  based  on  a  turboprop  engine  (the  content  was  intended  to  be  functionally  similar  to 
the  LM2500  marine  turbine  engine),  compared  the  two  groups  of  students  across  two  levels  of 
hypertext  control.  In  one  instance,  the  user  was  able  to  control  the  navigation  freely  without 
constraints.  In  the  other  condition,  the  user  was  provided  system  control  via  a  guide  window  that 
listed  specific  instructional  objectives,  and  at  the  users  selection  of  a  given  objective,  provided 
instructions  on  how  to  navigate  through  a  hypermedia  landscape  to  meet  that  objective. 

Using  a  2  X  2  analysis  of  variance  it  was  found  that  there  was  a  significant  interaction  of 
scores  between  the  level  of  prior  knowledge  and  the  level  of  control.  That  is,  the  low  prior 
knowledge  subjects  learned  more  under  system  control  but  high  prior  knowledge  subjects  learned 
more  under  user  control.  The  researchers  concluded  that  this  is  consistent  with  findings  of  other 
studies.  Inexperienced  learners  have  the  most  to  gain  from  system  guidance.  We  suspect  that  most 
learners  assigned  to  a  NAVEDTRACOM  course  could  be  called  low  prior  knowledge  learners  and 
that  the  need  for  guided  learning  applies. 

Another  study  looked  at  the  effect  of  some  different  but  related  learner  characteristics 
(Jacobsen  &  Spiro,  1995).  Specifically,  they  looked  at  the  influence  of  learning  in  a  hypertext 
environment  from  differences  in  learner  epistemic  cognition.  Episteraic  cognition  is  the  learners 
beliefs  about  learning.  They  include  what  the  learner  believes  are  good  and  bad  ways  of  how  to 
learn,  his/her  general  capability  to  learn,  the  nature  of  the  content  to  be  learned  and  the  learners 
ability  to  control  learning.  A  learners  epistemic  beliefs  were  suggested  to  be  associated  with 
whether  or  not  learners  can  integrate  new  knowledge  with  prior  knowledge  as  well  as  other  aspects 
of  learning. 

Epistemic  beliefs  and  preferences  (EBP)  was  measured  by  an  instrument  with  19  pairs  of 
statements  related  to  students  beliefs  about  learning.  Each  pair  had  a  positive  and  a  negative- 
worded  statement.  Learners  were  asked  to  agree  or  disagree  with  each  statement  on  a  7-point  Likert 
scale.  The  content  to  be  learned  was  the  impact  of  20th  century  on  society  and  culture.  The 
researchers  considered  this  domain  to  be  complex  and  ill-structured.  The  topic  had  many  concepts 
that  could  be  applied  to  dynamically  changing  real  world  events.  Three  forms  of  the  hypertext 
learning  environment  were  used  in  the  study:  (1)  a  minimal  hypertext  drill  similar  to  traditional 
computer-based-drill  programs  to  identify  a  single  them  about  the  topic,  (2)  minimal  hypertext  drill 
as  in  the  first  control  condition  but  with  the  requirement  to  identify  multiple  themes,  and  (3) 
thematic  criss-crossing  hypertext.  The  latter  was  the  experimental  condition  and  consisted  of 
noting  multiple  themes  through  criss-crossing  different  cases  that  had  been  previously  linked  based 
on  conditions  of  technology  and  societal  combinations. 

Measures  were  taken  from  transfer  tasks  including  essay  writing  on  the  topic.  Learners 
were  categorized  as  simple  EBP  and  complex  EBP.  A  simple  EBP  learner  preferred  simple  content 
and  memorization  of  pre-specified  content.  A  complex  EBP  learner  welcomed  working  with 
complex  knowledge  content  and  valued  his  /her  active  participation  in  the  learning  process.  It  was 
found  that  complex  EBP  learners  who  received  thematic  criss-crossing  hypertext,  rather  than  either 
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of  the  two  control  drill  conditions,  scored  higher  on  the  transfer  task  than  did  simple  EBP  learners. 
The  researchers  interpreted  this  to  be  related  to  simple  EBP  learners  having  had  difficulty  with  the 
nonlinear  and  multidimensional  aspects  of  the  hypertext.  We  think  that  these  findings  also  suggest 
additional  concerns  for  ensuring  that  hypertext  is  designed  to  support  the  learner,  not  job 
performance,  in  Navy  training.  While  the  job-aiding  lETM  is  designed  for  that  performance 
support,  learning  materials  should  be  designed  for  learning. 

There  are  many  variables  and  treatments  to  be  considered  in  hypertext/hypermedia  for 
learning.  We  have  only  touched  on  a  few  considerations  here  to  suggest  that  there  is  an  important 
difference  in  the  design  of  electronic  documents  for  achieving  goals  of  job  performance  support 
versus  learning.  Gall  (1994)  of  the  U.S.  Air  /Force  Academy’s  Center  for  Educational  Excellence 
has  proposed  one  framework  for  considering  the  pedagogical  considerations  in  design  of  hypertext 
learning  systems.  As  shown  in  Figure  2,  Galls  framework  relates  learner  attributes,  the  learning 
task,  the  educational  setting  and  the  hypertext  learning  system.  We  argue  that  each  of  these  should 
be  considered  independently  from  lETM  developments  at  the  present  time,  and  under  the  purview 
of  training  specialists,  until  more  experience  with  the  integration  of  lETMs  in  Navy  training  is 
available. 


Figure  2.  Relationships  among  hypertext  learning 
environment  components. 


5.2  Animation  and  Learning 

One  nontext  media  form  that  seems  to  be  talked  about  a  great  deal  for  lETMs  is  the  use  of 
animation.  Animation  is  certainly  an  advantageous  feature  possible  in  an  electronic  document  that 
is  not  available  in  paper  form.  There  are  also  probably  many  applications  for  it  that  are  meaningful 
for  supporting  job  performance  in  the  maintenance  of  complex  military  systems.  For  example, 
animation  could  be  used  to  show  a  part  on  a  piece  of  equipment  that  is  not  readily  visible  or 
identifiable  or  easily  discriminated  from  another  part  by  rotating  the  equipment  and  zooming  in  to 
show  its  location  and  contrast  it  to  other  locations. 
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Animations  to  be  used  in  learning  environments  have  additional  instructional 
considerations.  A  recent  comprehensive  review  of  literature  (Wetzel,  Radtke,  &  Stem,  1993) 
regarding  instructional  animations  has  surfaced  some  of  the  main  aspects  to  be  considered.  They 
concluded  that  the  effectiveness  of  instractional  animations  has  been  shown  more  with  younger 
learners  (e.g.  grade  and  high  school  students)  than  with  adults.  Of  16  studies  with  adults  that 
compared  the  effectiveness  of  animations  against  non-animation  techniques,  such  as  static 
graphics,  only  three  produced  positive  results.  Part  of  the  reason  for  this,  however,  may  be  related 
to  how  the  animation  was  used.  When  animation  is  used  by  itself  to  present  content  it  was  seen  to 
be  relatively  less  effective  based  on  the  studies  reviewed.  When  it  is  used  with  practice,  probably 
with  direct  manipulation  interactions  so  that  learners  can  see  changes  occur  in  direct  and 
continuous  response  to  their  actions,  learning  can  be  effective. 

According  to  a  history  of  direct  manipulation  written  by  Hutchins,  Hollan  and  Norman 
(1986)  there  are  three  properties  of  direct  manipulation: 

1.  Continuous  visual  representation  of  the  object  of  interest  in  relationship  to  user  actions. 

2.  Physical  actions  or  labeled  button  presses  which  can  be  taken  by  the  user  instead  of  com¬ 
plex  syntax  commands. 

3.  Rapid,  incremental,  reversible  operations  whose  impact  on  the  object  of  interest  is  imme¬ 
diately  visible. 

An  example  of  this  approach  familiar  to  Navy  training  personnel  (and  developed  initially 
by  Hollan  and  colleagues)  is  the  Steamer  Intelligent  Tutor  developed  at  NPRDC  in  the  mid  to  late 
1980s.  As  stated  by  Hutchins,  et  al  (1986)  we  see  promise  in  the  notion  of  direct  manipulation,  but 
as  yet  no  explanation  for  it  (p.92).  These  authors  go  on  to  say  that  examples  of  this  technique  seem 
to  feel  natural  to  users  because  they  also  provide  a  feeling  of  directness.  They  then  go  on  to  develop 
a  psychological  framework  which  defines  two  aspects  of  directness;  distance  and  engagement. 
Distance  refers  to  the  space  perceived  by  a  user  between  his/her  thoughts  and  the  physical 
requirements/state  of  the  computer.  Engagement  refers  to  the  perception  by  the  user  that  he/she  is 
directly  manipulating  the  objects  of  interest  by  the  actions  being  taken. 

This  theoretical  notion  is  consistent  with  conclusions  reached  about  instructional 
animations  by  Rieber  and  Kini  (1991).  They  concluded  that  the  primary  advantage  of  animations, 
over  static  graphics,  is  the  visual  representation  of  change  or  movement  over  time.  They  also 
suggest  that  the  animation  enhances  the  learners  ability  to  perceive  apparent  motion  and  thereby 
lessens  the  cognitive  load  in  learning,  freeing  up  the  learners  resources  to  attend  to  other  aspects 
of  the  learning  task.  Rieber  (1990)  made  three  design  recommendations  for  the  use  of  instructional 
animation: 

1.  Animation  should  be  incorporated  only  when  its  attributes  are  relevant  to  the  learning 
task.  These  attributes  are  visualization,  cognitive  meaning,  and  motion. 

2.  Since  novices  may  not  know  how  to  attend  to  relevant  cues  provided  in  the  animation, 
learning  should  be  packaged  to  include  text,  static  graphics  or  other  forms  along  with  the  anima¬ 
tion. 
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3.  The  power  of  animation  for  computer-based  instruction  may  be  in  interactive  graphic 
applications. 

We  see  these  conclusions  as  consistent  with  the  concept  of  why  direct  manipulation 
interfaces  are  of  value,  but  we  extend  the  requirements  for  instruction  to  include  multiple  forms  of 
support  to  the  learner  through  text  explanations,  still  graphics,  and  instructors  themselves. 
However,  not  all  students  are  successful  in  all  situations  learning  this  way  and  the  exact 
composition  of  animations,  text,  narration,  and  other  visuals  is  still  under  study.  Research  is  in 
progress  to  better  define  when  to  use  this  form  of  computer  user  interface.  For  example,  some  of 
the  research  to  date  indicates  that; 

•  When  a  verbal  description  of  a  simple  operation  (using  a  bicycle  tire  pump)  was  given 
either  before  or  during  the  presentation  of  an  animation  the  during  group  outperformed 
the  before  group  on  a  related  problem-solving  test  (Mayer  &  Anderson,  1991). 

•  When  a  verbal  description  of  the  bicycle  tire  pump  operation  was  presented  with  no 
animation,  a  narration  without  the  animation,  and  a  verbal  description  during  the  ani¬ 
mation,  the  during  group  again  outperformed  the  others  (Mayer  &  Anderson,  1991). 

The  latter  results  were  interpreted  by  the  researchers  to  support  a  theory  that  there  are  two 
kinds  of  memory  coding  related  to  these  two  different  forms  of  information.  One  was  termed 
representational  connections  and  occurs  between  either  verbal  stimuli  and  verbal  representations 
or  between  visual  stimuli  and  visual  representations.  The  other  is  called  referential  connections  and 
occurs  between  visual  and  verbal  representations.  The  researchers  concluded  that  the  results 
support  a  view  that  students  use  both  words  and  pictures,  such  as  animations,  to  build  an  effective 
mental  representation  of  how  a  system  works  through  referential  connections.  Verbal  and  visual 
presentations  should  be  presented  together  in  order  for  students  to  make  referential  connections 
and  for  more  powerful  learning  to  occur.  Presenting  verbal  and  visual  explanations  without 
connecting  them  is  not  as  helpful  as  connecting  them. 

We  suggest  that  this  may  also  relate  to  why  a  direct  manipulation  is  so  effective.  The 
instructor  normally  explains  what  is  happening  when  this  technique  is  used  for  a  demonstration. 
When  a  student  uses  it,  however,  he/she  must  generate  the  explanation  themselves  for  it  to  be 
effective  unless  we  provide  explanations  simultaneously  to  the  student  and  ensure  the  connection 
is  made  instead  of  letting  it  be  discovered.  Thus,  Navy  training  could  be  served  well  to  provide 
animated  simulations,  in  a  direct  manipulation  mode,  for  both  demonstration  with  instructor 
explanations,  and  as  stand-alone  devices  for  students  as  long  as  concurrent  explanation  can  be 
provided.  This  may  be  very  different  from  the  lETMs  animations  used  for  job  support. 

5.3  Motion  Video  and  Learning 

Video  sequences  that  demonstrate  procedures  to  be  performed,  show  the  characteristics  of 
equipment  in  motion,  or  in  some  other  way  depict  movements  that  the  lETM  user  needs  to 
discriminate  have  also  frequently  been  discussed  by  lETM  developers  as  desirable  aids  to 
performance  support.  There  also  seems  to  be  an  implication  that  these  videos  are  instructional  and 
could  carry  instructional  value.  We  again  caution  against  the  assumption  that  the  motion  video  used 
as  a  job  aid  is  the  same  as  one  used  for  instruction.  Just  as  described  for  animation  above,  the 
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instructional  characteristics  of  effective  motion  video  are  dependent  on  student  attributes  such  as 
prior  knowledge  and  level  of  experience;  breadth,  depth  and  complexity  and  of  the  content;  the 
instructional  setting  including  availability  of  an  instructor  or  other  experienced  person  who  can 
coach  and  explain;  and  context  with  other  media  including  text  or  media  that  supplement  views  of 
the  video. 

Wetzel,  et  al.  ( 1993)  recently  reviewed  studies  of  video  and  film  motion  used  for  procedural 
learning.  As  described  by  those  authors,  motion  is  widely  thought  to  be  effective  as  a  visualization 
technique  in  instruction.  Benefits  attributed  to  video  motion  in  the  research  literature  include  its 
capability  to  show  the  motion  or  other  visual  information  itself,  visually  present  discriminations 
needed,  gain  attention,  depict  continuity  in  sequential-spatial  relations,  or  represent  information 
that  is  difficult  to  fully  describe  verbally.  One  interesting  conclusion  reached  by  these  reviewers 
was  that  the  studies  clearly  showed  a  powerful  benefit  to  result  from  the  addition  of  hands-on 
practice...  (p.63).  This  implies  to  us  the  need  to  design  an  instructional  setting,  perhaps  in  a 
laboratory  situation,  where  such  practice  can  be  available.  In  addition,  the  motion  sequences  found 
to  be  effective  in  comparison  to  other  techniques,  such  as  static  graphics  or  text,  often  only  depicted 
selected  aspects  chosen  for  pedagogical  reasons.  The  designers,  for  example,  specifically  used 
motion  video  or  film  to  enhance  instructional  effectiveness  by  showing  features  for  critical 
discriminations. 

There  are  many  considerations  in  making  a  video,  whether  for  job  performance  support  or 
instruction  and  this  is  not  the  place  to  cover  them  in  detail.  We  do,  however,  hope  that  the  point  is 
made  that  an  lETM  developed  for  effective  job  performance  support  cannot  be  expected  to  be  the 
same  vehicle  for  effective  instruction  even  if  it  incorporates  good  videos,  animations  and  other 
media. 
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Section  Six 


Conclusions 


6.0  Conclusions 


In  summary,  we  offer  the  following  conclusions. 

6.1  Preparing  and  Supporting  the  IVaining  Setting  for  lETMs 

In  Section  4  we  discussed  the  lessons  learned  and  issues  that  have  arisen  when  lETMs  are 
introduced  into  the  classrooms  and  laboratories.  A  major  impact  is  that  there  may  be  refurbishment 
needs  for  the  physical  facilities,  such  as  air  conditioning  and  power,  and,  of  course,  the  acquisition 
and  continuing  support  of  the  lETM  computers.  These  are  important  considerations  that  require 
planning  and  budgeting  well  in  advance  of  the  need  to  conduct  lETM-based  training. 
Organizations  such  as  the  NAVSEA  and  CNET  are  taking  leadership  roles  in  their  areas  of 
responsibility  in  the  Navy  to  provide  an  adequate  basis  for  implementation  and  support  in  the 
future.  However,  the  procedures  and  techniques  are  still  new  and  can  be  expected  to  be  fine  tuned 
later.  Certainly,  an  important  constraint  is  time.  The  normal  planning  time  needed  for  budgeting  in 
training  may  be  a  problem  to  integrate  the  number  of  lETMs  that  will  be  implemented  in  the  near 
future. 


6.2  Changing  Nature  of  lETM  Electronic  Classrooms 

In  Section  4  we  pointed  out  that  the  introduction  of  computers  into  the  classrooms  and 
laboratories  to  support  instruction  using  lETMs  parallels  the  movement  to  implement  electronic 
classrooms.  To  some  extent,  the  current  mental  model  of  the  classroom  as  an  instructor-led,  on-the- 
podium,  group  learning  setting  is  now  being  simply  transferred  over  to  the  lETM  based  electronic 
classrooms.  We  do  not  argue  against  this  as  a  first  step,  but  suggest  that  changes  in  student  learning 
and  the  instructor  roles  may  be  worth  considering  in  the  next  few  years.  It  is  suggested  that  not  only 
can  the  instructors  role  change,  but  so  can  the  students.  The  lETM-  based  electronic  classroom  can 
be  structured  for  students  to  take  more  responsibility  for  themselves  and  with  each  other. 

In  addition,  it  can  be  expected  that  the  definition  of  the  classroom  will  continue  to  change 
as  its  functionality  is  enhanced  by  the  new  technology  available.  The  virtual  classroom  may  be  one 
of  those  ways  it  changes.  The  networked  virtual  classroom,  whether  for  physically  co-located 
groups  or  physically  separated  individuals,  offers  the  promise  of  distributing  the  instruction  to 
more  people  with  less  instructor  resources. 

6.3  lETMs  in  On-Demand  and  Just-In-Time  Training 

As  also  discussed  in  Section  4,  the  lETM-based  training  is  being  projected  to  result  in  fewer 
courses  and  less  training  time.  Several  military  agencies  responsible  for  the  implementation  of 
lETMs  and  training  are  studying  the  option  to  move  more  training  out  of  the  classroom  by 
supporting  performance  in  the  field  with  smart  lETMs  and  computer-based  training  (CBT)  that  can 
be  accessed  as  needed.  Although  no  organization  we  surveyed  was  considering  total  elimination  of 
classrooms  and  associated  laboratories,  especially  for  initial  training  of  apprentices,  there  was  a 
definite  inclination  to  reduce  the  formal,  centralized  school  time.  This  initiative  revolves  around 
the  concept  of  just-in-time  training  with  a  performance  support  system.  Just-in-time  training  seems 
to  be  a  focus  especially  for  the  military  and  industry  areas  of  aircraft  maintenance. 
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6.4  lETMs  as  Expert  Job  Aids 


As  described  in  Section  4,  several  of  the  organizations  interviewed  expressed  strong 
preference  for  moving  the  capability  to  support  the  technician  through  more  use  of  lETMs  and  less 
use  of  training.  The  more  intelligence  that  can  be  put  into  an  lETM,  the  better  the  operators  and 
maintainors  of  equipment  can  be  supported  to  fix  what  breaks.  This  is  certainly  a  redefinition  of 
jobs  and  traimng  requirements  as  a  result  of  lETM  technology.  However,  we  note  that  not  all 
BETMs  will  have  such  a  capability.  The  cost  of  developing  class  4  and  5  lETMs  versus  class  3 
lETMs  may  be  a  stumbling  block.  The  economics  of  lETMs  is  beyond  the  scope  of  this  report,  but 
we  hesitate  to  suggest  that  smart  documents  can  be  built  in  the  near  future  to  impact  a  wide  number 
of  jobs.  Many  lETMs  for  mature  systems  will  lack  the  smart  performance  support  features  and  will 
not  be  integrated  into  a  total  maintenance  system. 

6.5  Implications  for  the  Organizations 

The  integration  of  lETM  technology  into  training  and  education  requires  a  look  at  the 
coordination  roles  and  functions  of  organizations.  There  appears  to  be  evidence  that  training  and 
documentation  organizations  will  need  to  change  at  several  levels.  Typically  in  the  past,  large 
organizations  inside  and  outside  the  military  have  had  one  unit  responsible  for  documentation  and 
another  one  responsible  for  developing  and  operating  training.  Because  of  the  impact  on  both 
areas,  it  will  be  necessary  for  even  greater  coordination  between  the  agencies  and,  most  likely,  an 
integration  of  the  agencies  at  least  for  the  purpose  of  developing  and  managing  lETMs.  This  will 
be  especially  true  if  the  path  of  an  integrated  data  base  is  taken. 

6.6  lETM  Data  as  a  Source  for  Developing  Training 

As  discussed  in  Section  4,  one  way  to  think  of  lETMs  is  as  a  source  of  data  for  developing 
training  materials.  Several  of  the  organizations  surveyed  pointed  out  that  the  lETM  data  base  of 
technical  information  contains  the  same  information  needed  to  produce  CBT.  In  fact,  it  is  typical 
when  developing  any  instructional  material  of  a  technical  nature  to  draw  from  the  job  documents, 
when  they  are  available,  for  source  data.  Similarly,  lETM  end  user  organizations  in  the  training 
world  hope  that  CBT  can  be  developed  with  automated  aids  from  an  lETM  database.  Efforts,  like 
that  by  the  Air  Force  Armstrong  Laboratory  with  XAIDA,  to  automatically  generate  CBT 
courseware  from  information  input  from  lETM  database  elements  may  save  a  great  deal  of  time 
and  money  and  should  be  continued.  The  Navy  AIM  I  and  II  developments  may  also  be  pursuing 
similar  objectives. 

6.7  Common  Data  Elements  and  Integrated  Databases 

Given  the  cost  of  data  development  for  both  manuals  and  training,  and  the  commonality  of 
information  used  in  both  applications,  it  seems  reasonable  to  consider  the  possibility  of  defining 
the  data  elements  so  that  such  diverse  applications,  as  well  as  other  logistics  applications,  could 
reuse  the  same  data  with  ease  and  efficiency.  The  goals  would  be  to  develop  data  once  and  to  define 
it  in  such  a  way  as  to  facilitate  electronic  data  interchange  among  applications.  There  are  several 
movements  in  progress  that  make  this  more  feasible  now  than  in  previous  years.  However,  not  all 
of  the  pieces  are  in  place.  In  Section  4  we  suggest  that  it  is  well  worth  studying,  perhaps  through  a 
joint  service,  multi-disciplinary  committee,  the  possibility  of  achieving  a  common  database  and 
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leveraging  the  efforts  underway  with  MIL-STD-1379D  to  influence  the  possible  accomplishment 
of  the  goal. 

6.8  lETM  Technology  in  Education 

In  education  the  goal  is  often  to  teach  students  how  to  access  information,  determine 
relationships,  use  information  to  solve  problems,  and  develop  multiple  perspectives  on  the 
information.  BETM  technologies  can  be  used  to  support  such  goals  in  education.  As  we  discussed 
in  Section  3.3,  we  see  similar  use  of  SGML  and  lETM  technologies  for  case-based  education  and 
training  in  the  mihtary.  There  are  many  military  instructional  situations  where  it  is  more 
appropriate  for  the  events  of  learning  to  be  adult,  learner-centered  rather  than  instructor-  centered. 

6.9  Embedding  Training  Within  lETMs 

In  Section  5  we  expressed  concerns  about  including  training  in  lETMs  based  only  on  the 
same  technical  information  required  for  job  performance  support.  We  attempted  to  make  a  case  to 
show  that  there  is  a  gap  in  the  understanding  of  some  people  in  the  lETM  development  world  of 
how  training  materials  are  designed  to  meet  goals  for  learning.  It  is  our  contention  that  while 
lETMs  and  training  need  to  be  integrated  it  must  be  done  so  with  full  partnership  of  those  in  the 
training  community  who  can  provide  expertise  in  learning  with  the  technology  of  hypertext/ 
hypermedia,  and  new  digital  media  such  as  animations  and  video. 

We  have  also  raised  questions  about  roles  and  authority  of  organizations  if  instruction  is 
included  embedded  in  lETMs.  For  example,  who  will: 

•  Define  it  as  a  requirement  and  continue  to  assess  that  requirement. 

•  Design  it  for  effectiveness,  taking  into  account  differences  in  target  population  charac¬ 
teristics. 

•  Design  it,  taking  into  account  media  selection  principles. 

•  Design  it,  taking  into  account  instructional  strategies  for  different  types  of  learning 
and  learners. 

•  Manage  the  configuration  from  a  basis  of  both  instructional  and  job  performance 
effectiveness. 

•  Manage  the  continuing  support  required  for  the  combinations  of  classroom,  fleet  and 
just-in-time  training. 

It  is  apparent  that  these  are  issues  that  have  not  yet  been  addressed  fully  at  top  level  training 
management  levels.  If  training  is  to  move  more  from  the  centralized  classroom  to  the  field  via 
lETM  media,  then  we  recommend  a  more  active  role  on  the  part  of  organizations  such  as  the  Naval 
Education  and  Training  Command,  Air  Force  Air  Education  and  Training  Command,  and  Army 
Training  and  Doctrine  Command. 
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6.10  Beyond  lETMs  and  SGML 


The  technology  underlying  lETMs—  from  computers  to  communications  to  displays  to 
databases~has  changed  dramatically  over  the  ten  years  or  so  that  the  military  has  been  working  at 
electronic  technical  manuals  developments.  There  would  have  been  no  way  to  predict  the 
directions  of  that  technology  except  to  expect  rapid  advancement.  The  next  five  to  ten  years  will 
continue  to  bring  evolutions  in  these  and  other  related  technology  areas.  That  we  are  in  motion  and 
not  done  yet  is  perhaps  self  evident,  but  worth  stating  anyhow.  While  we  must  proceed  ahead  to 
achieve  gains  in  effectiveness  of  job  performance  and  military  readiness  through  lETMs, 
integrated  properly  with  training  and  education,  we  should  do  so  with  a  wary  eye  toward 
determining  the  best  way  to  do  so  across  time. 

We  can  only  speculate  what  some  of  these  technology  evolutions  will  be  by  talking  about 
what  is  happening  today.  Certainly  we  can  expect  in  the  next  few  years  to  see  greater  use  of 
hypermedia  including  media  elements  such  as  animations  and  video.  Better  hardware  and  software 
techniques  for  handling  and  generating  media  elements  will  provide  more  cost  effective 
accessibility  with  greater  capabilities  even  within  the  next  year.  As  pointed  out  by  Hanlon  (1995b), 
lETM  lessons  learned  recently  have  suggested  the  need  for  incorporating  HyTime  as  a  software 
neutral  way  to  capture  lETM  intelligence  and  Metafiles  for  Interactive  Documents  (MID)  to  solve 
problems  of  lETM  presentation  portability.  We  can  expect  to  see  increased  bandwidths  for 
distribution  of  large  amounts  of  lETM  type  documents  as  the  infrastructure  for  the  information 
superhighway,  on-demand  television  programming,  and  related  applications  becomes  real.  We  can 
expect  to  see  more  development  of  integrated  databases  and  functionality  as  market-driven  open 
standards  such  as  Object  Linking  and  Embedding  (OLE),  Open  Database  Connectivity  (ODBC), 
and  Standard  Generalized  Markup  Language  (SGML)  are  applied  and  grow.  SGML  is  already 
being  extended  to  enhance  its  applicability  to  such  events  as  the  World  Wide  Web  and  multimedia 
documents.  There  are  also  several  market  driven  directions  that  display  technology  is  headed  for 
including  lighter,  flatter,  and  higher  resolutions.  Finally,  even  now  there  are  considerations  in  front 
of  the  DoD  CALS  committee  that  could  change  the  nature  of  lETMs  in  the  future.  We  can  at  the 
least  see  better  portability,  increased  functionality,  and  more  integration  with  other  applications. 
The  changes  to  lETM  classification  definitions  are  not  done.  If  lETMs  as  we  know  them  in  their 
state  today  can  be  expected  to  change  so  too  can  the  impact  on  both  training  and  education. 
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Section  Seven 
Standards  and  Specifications 


7.0  Standards  and  Specifications 


American  National  Standards  Institute.  Information  Processing-Text  and  Office 
Systems-Standard  Generalized  Markup  Language  {SGML}.  (ISO/IEC  8879),  1992. 

American  National  Standards  Institute.  Information  technology-Hypermedia/Time  based 
Structuring  Language  (Hytime).  (ISO/IEC  IS  10744),  1992. 

DoD  Directive  8320.1.  DoD  Data  Administration.  26  September  1991. 

DoD  Directive  8320. 1-M.  Data  Administration  Procedures.  March  1994. 

DoD  Directive  8320. 1-M- 1.  Data  Element  Standardization  Procedures.  January,  1993. 

Military  Specification  MIL-D-87269.  Data  Base,  Revisable:  Interactive  Electronic 
Technical  Manuals,  for  the  Support  of  20  November  1992. 

Military  Specification  MIL-M-87268.  Manuals,  Interactive  Electronic  Technical:  General 
Content,  Style,  Format,  and  User-Interaction  Requirements.  20  November  1992. 

Military  Specification  MIL-Q-87270.  Quality  assurance  Program:  Interactive  Electronic 
Technical  Manuals  and  Associated  technical  Information;  Requirements  for. 
November,  1992. 

Military  Standard  MIL-STD-1379D.  Military  Training  Programs.  5  December  1990. 

Naval  Education  and  Training  Command  NAVEDTRA  135.  Navy  School  Management 
Monua/.  September  1992. 
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